Managing IT data is maddening. IT organizations are hamstrung right and left by lack of complete and high quality information, yet approaches to resolve this have been only partially successful – at best!
One approach that has had less attention is the idea of a data warehouse for IT data. As big data and analytics get more and more traction, the idea of applying them to IT is obvious.
Now, a CIO, if presented with the idea of an “IT data warehouse,” may respond “I thought we had one of those – the CMDB (Configuration Management Database, or System).” But the CMDB/CMS does not fit the classic data warehouse definition in important respects, especially in that data warehouses are typically read-only (non-volatile).
In reality, the CMDB has become more of a transactional back-end for production operations. The “CMDB” found at the heart of modern IT service management systems (such as BMC Atrium) may have extensive integrations, but this is not sufficient to consider it a data warehouse.
Often, a CMDB accepts direct updates of its data. The needed reporting “dimensions” may not exist in the CMDB. Although it may have audit trails, it typically is not rigorously historical in the sense of a true data warehouse. If the CMDB is running on the same system supporting high volume Incident and Service Desk processes, long running analytic queries might seriously hamper performance.
All in all, the CMDB is a poor analytic platform.
In data warehousing terms, the CMDB is more akin to what Bill Inmon termed the “Operational Data Store” or ODS . The ODS, while it integrates data, is current state oriented and can be transactional. An ODS may feed an enterprise data warehouse, but should never be confused with one. Similarly, the CMDB may feed the IT data warehouse, but should not be confused with it. (In fact, a CMDB is not required to have an IT data warehouse.)
An IT data warehouse can be assembled with off the shelf tools, but this is labor intensive and vendor offerings can add substantial value. A number of players are offering solutions that are not CMDBs. Rather, these products draw from the CMDB/CMS as one of various sources:
- HP continues to invest in the warehouse capability underlying its IT Executive Scorecard and IT Financial Management solutions (EMA white paper).
- SAS also continues to evolve its IT Resource Management solution, which seeks to provide value for both resource/capacity and IT financial management.
- Blazent solves difficult problems with IT asset and inventory reconciliation via integrating and analyzing a multitude of sources.
- PureShare offers its SingleView IT operational IT performance reporting, integrating sources such as service desk and IT financial management
- Apptio has a powerful data profiling and reconciliation architecture it employs in its Technology Business Management solution.
- Emerging player Northcraft Analytics has offerings for both Remedy and ServiceNow metrics and KPIs.
These solutions are more at the “business of IT” level. At the element management and in the governance, risk and compliance domains we see many more interesting and specialized data aggregators and analytics engines, such as Splunk, Arcsight, Rev2, Prelert, and many more. See also Watson joins the IT management team.
As data warehousing is an extensible and scalable approach, many of these vendors are well positioned to move from their origins into more cross functional problems. An integrated data architecture has powerful network effects, as each new integrated source enables an exponentially increasing number of analysis use cases.
What’s the downside? Services, in a word. These are not, and never will be, plug and play solutions. Sourcing, extracting, transforming, and loading the data takes investment, especially from lesser known or custom built sources. You may spend as much on development as you do on acquisition. You might be tempted to do it all from scratch, but be careful there; sound data warehouse architectures take a lot of thought. (How will you handle slowly changing dimensions?)
Furthermore, the very term “data warehouse” may have negative connotations in some organizations, as an overly heavyweight, expensive proposition. Proposing a data warehouse on the heels of a failed CMDB implementation will probably not win you any friends. But data warehousing remains a vital and growing practice area in the overall IT landscape. When carefully considered and executed, it’s led to many successes. Just go to a TDWI course or two.
As always, thoughts appreciated.
