Blog: Claudia Imhoff Subscribe to this blog's RSS feed!

Claudia Imhoff

Welcome to my blog.

This is another means for me to communicate, educate and participate within the Business Intelligence industry. It is a perfect forum for airing opinions, thoughts, vendor and client updates, problems and questions. To maximize the blog's value, it must be a participative venue. This means I will look forward to hearing from you often, since your input is vital to the blog's success. All I ask is that you treat me, the blog, and everyone who uses it with respect.

So...check it out every week to see what is new and exciting in our ever changing BI world.

About the author >

A thought leader, visionary, and practitioner, Claudia Imhoff, Ph.D., is an internationally recognized expert on analytics, business intelligence, and the architectures to support these initiatives. Dr. Imhoff has co-authored five books on these subjects and writes articles (totaling more than 150) for technical and business magazines.

She is also the Founder of the Boulder BI Brain Trust, a consortium of independent analysts and consultants ( You can follow them on Twitter at #BBBT

Editor's Note:
More articles and resources are available in Claudia's BeyeNETWORK Expert Channel. Be sure to visit today!

April 2005 Archives

There have been a number of reports recently about security and privacy breaches involving customer data -- ChoicePoint, Lexis-Nexis and other information providers come to mind. What do you need to do to ensure that this never happens to your corporation's data?

I have been reading a lot lately about privacy and the need to secure customer data from hackers and outsiders who want to misuse the information. The erosion of privacy even within corporations seems to be pretty concerning as well. What is a corporation to do to ensure that its crown jewels -- its integrated customer data -- does not fall into the hands of the wrong people?

It was suggested to me recently that the only way to really secure this data is to encrypt it. If hacked into, the hackers would only get a string of gibberish and nothing meaningful would be usable. The only people who could use the data would be those individuals having the encryption key. Sounds good but there are problems with this approach too.

Encryption traditionally has involved a substantial amount of overhead -- overhead to encrypt the data, thus slowing down the data processing (think loading of data into the data warehouse, for example). Then there is the additional overhead of decrypting the data when someone wants to use it. There is good news here though. Encryption (and the corresponding decryption) technologies have come a long way. Encryption vendors now claim the ability to encrypt and decrypt with little or no performance hits whatsoever.

The second thing about encryption -- particularly from a BI point of view -- is that once it is decrypted and the query returned to the user, the data may be downloaded to their PC. Hmmm -- where's the security now? Unless it stays encrypted -- even on the user's PC, it seems to me that we still have a major hole in the overall data security and privacy scenario. This is an area that could use some vendor support as well.

If you have any ideas about security and data privacy, I welcome your comments.

Yours in BI success,


Posted April 28, 2005 10:46 AM
Permalink | 1 Comment |

One of the questions that I get quite often is "Do I need to create an enterprise data model?" In the case of a recent phone call, the caller stated that his company never built applications; they only bought off the shelf software so why would he need such a data model. Here are my reasons for creating the model despite the buy versus build philosophy.

The caller's company had bought a set of a large ERP vendor's modules. Now they were looking at another ERP vendor for a new module. They also purchases CRM applications and even BI ones. Why should his group worry about an enterprise data model (EDM)? I told him that basically there were two main reasons (there are other smaller ones but let's stick with the biggies):

1. The purpose of the EDM is to capture or document the "ideal state" of the corporation's data needs. In other words, what data is needed (the attributes and entities of interest to the company), how it is associated with other data (the important relationships between entities), the business rules behind the associations (is it a mandatory or optional relationship?), and so forth. When buying a COTS piece of software, you would use your data model to determine how closely the application will match to your ideal state. The analysis should determine where there is a perfect match and the software will support your needs exactly -- where the application is close but not perfect and some modification or tweaks to the software may be needed - and where there is no match at all and either the corporation must change the way it currently does business or look for another application that is closer to their ideal. Without the EDM, I don't know how you can make such an assessment of fit.

2. Since no company seems to have a fully integrated set of operational systems (even these folks had different ERP and CRM vendors involved), some sort of "ideal" or master set of translation tables will be indeed. The EDM can serve that purpose as well. With the EDM in hand, you have a means of mapping one application to the other using the EDM as "translation tables" to ensure smooth integration. The definitions of the data, documented relationships between the data, and so on, will ensure that a proper integration can occur.

Hopefully these thoughts will help you justify the need for an EDM in your own organization. As always, I welcome your input and experiences along these lines.

Your in BI success,


Posted April 26, 2005 10:29 AM
Permalink | 6 Comments |

An excerpt from the Open Source Initiative explains why open source is important to the technologists. They state "The basic idea behind open source is very simple: When programmers can read, redistribute, and modify the source code for a piece of software, the software evolves. People improve it, people adapt it, people fix bugs. And this can happen at a speed that, if one is used to the slow pace of conventional software development, seems astonishing."

So what does this mean to you?

Yesterday I spoke with CEO Sam Mohamad and CTO Luke Lonergan from Greenplum (formerly Metapa), a company that is a big proponent of open source technology for BI, about open source and the role it will play. Here are some excerpts from that conversation:

CMI: Why is open source important in today's BI market?

Greenplum: The current applications development community favors open source because it enables them to try a greater breadth of options more quickly, which injects more innovation into the process. Given the increasing stability and features of the open source offerings in many applications, business managers are becoming accustomed to the innovation and cost-effectiveness of the model.

CMI: Why does it have more appeal for larger corporations than smaller ones?

GP: Large corporations traditionally have had more difficulty in innovating than small ones. Open source is changing the model of innovation by putting more power into the hands of the developers and departmental leaders building the newest revenue-producing products. The risk of adoption is being spread over a greater number of applications, and the best are bubbling up with the best-of-breed open source foundations. Companies like RedHat have prospered from this trend.

CMI: What are the disadvantages of open source technologies to corporations?

GP: One of the big drawbacks to adoption of open source tools is that there isn't a clear picture of how the tool will evolve over time. As a result, there is less predictability of feature and function in the open source tool sets.

What mitigates this problem is the focusing influence of companies that commercialize open source tools by concentrating the community on the most relevant feature sets for chosen audiences. Combining this focus with the appropriate support infrastructure makes the packaging more comfortable and predictable the enterprise.

Posted April 22, 2005 3:16 PM
Permalink | 2 Comments |

I can't tell you how many times BI designers have told me that their users want to see reports filled with averages -- average sales by store, average customer purchase, average inventory levels, and so on. Do people really understand how misleading and erroneous these figures can be? Here's my tip for you.

Average calculations are "smoothing" techniques; they remove the ups and downs of the actual data. They may be useful to give you a high level trend or estimation but they can be completely misleading if you base business decision on them. Let me give you an example of what I mean.

Suppose someone in charge of inventory levels asks for the average monthly sales for product A. The answer comes back that, on average, the company sells 100 units of Product A per month. Does that mean that there should be only 100 units and no more of Product A in inventory every month? I wouldn't bet my job on that...

The average is based on the fact that 1200 units of Product A sold over a 12 month period. This does not take into account the fact that the product's sales are seasonal, affected by marketing campaigns, or boosted by recurring external factors like sports events. In reality, Product A is quite seasonal with 80% (or 960 units) of its sales occurring in the summertime -- a three month period. In fact, the units may sell withing a 6 week period within even the 3 months. The rest are sold on either side of summer with no sales occurring in winter.

If the inventory were stocked at 100 units per month, there would not be enough available in the height of summer to sell so sales would be lost. Yet, the levels would be way too high in winter, causing unnecessary inventory carrying costs. That is the problem with averages...

In gathering requirements from your business users, make sure you dig a bit deeper into the users' needs than hearing just a cursory "I want the average sales amounts of..." It is always useful to ask why and how the data be used. It is wise to store the details that go into each of these average amounts in the warehouse to ensure that it remains possible to look at the actual numbers as well as averages.

I hope you find this tip useful. If you have others, please add them to this blog.

Yours in BI success,


Posted April 20, 2005 3:15 PM
Permalink | 2 Comments |

In a recent Conde Nast Traveler Magazine (March 2005), they reported on their readership poll (designed with Carnegie Mellon’s Risk Perception and Communication and Harvard’s Center for Risk Analysis) that was aimed at the most fundamental travel question – to go or not to go. Their analytic results were surprising in some respects.

The results show that although most of us may be debating this question, we are largely choosing to take the chance and go. What they deemed as remarkable is that we are NOT taking seriously the travel advisories and warnings issues by the US State Department and the Department of Homeland Security.

OK – maybe I have been a road warrior for way too long but haven’t we been under a code yellow or orange travel alert for more than three years? Does it surprise you that we travelers are no longer affected by these warnings? That we feel these warnings are influenced more by politics than reality? More by business interests and legal liabilities than real danger to the traveling public?

In short, the poll demonstrated that we don’t trust our government’s security information or analytics. It desperately needs a reliable BI environment. Our government must hire the best of the best BI consultants to get its analytics straightened out and produce reliable, consistent, nonpolitical intelligence about our travel situation…

Yours in BI success,


Posted April 18, 2005 9:43 AM
Permalink | 2 Comments |

Did you know that the Internal Revenue Service (IRS) has one of the most sophisticated analytical environments to detect tax payer fraud? Yep, it's true according to the technology company supporting it...

Since today is tax day across America, I thought it would be appropriate to let you know what the IRS has -- just in case you thought about fudging a few of your numbers... The Research Department of the IRS has created a single integrated environment using Sybase IQ technology. Yikes!

According to the case study, this data management and analytics system was built for the IRS user community to provide fast, flexible analysis. This "Compliance data warehouse" (CDW)holds seven years worth of business and individual tax return raw data, amounting to 12 TB of input data (compressed at a 70% compression rate with IQ). Now these users can slice and dice, look for fraud, analyze trends in tax returns to their hearts' content. Pretty scary, huh?

And today they are about to get another 1.7 TB of tax information!

Sigh -- yours in BI success...


Posted April 15, 2005 8:34 AM
Permalink | No Comments |

Recently I spoke with Doug Laney (formerly an analyst with META Group and now CEO and Chief Research Officer of Evalubase Research) about the results of his company’s latest research results. I found the BI results particularly intriguing.

Evalubase has a very creative and different method for getting technology evaluations. The company surveys a variety of IT professionals throughout our industry continuously about their real-life experiences with solutions. They gather a multitude of hard metrics and several free-form, unstructured comments in each completed evaluation. You, the participant, can choose to evaluate any technology you like whenever you like, and answer any questions you feel qualified to do so.

Since this is an ongoing study of the IT market, you will also find an “aging” algorithm to mitigate the impact of older evaluations. Let’s face it – vendors improve or get worse over time. Therefore, to reduce the undue effects of old evaluations, recent evaluations always count more heavily in the calculation of a metric or rating.

That said – what did they find (as of April 11, 2005)?

1. ERP applications won overall approval in terms of functionality, efficiency and reliability. CRM came in second. BI came in fourth – possibly suffering from vendor hype.

2. For overall ingenuity, Siebel got top honors (though, interestingly, they were last in terms of their credibility…), followed by HP, Microsoft, IBM and Oracle. SAP was ranked last in the line up.

3. In terms of overall key buying criteria for enterprise solutions, buyers were fairly consistent in ranking all criteria fairly equally – ease of integration, product performance, and functionality were slightly ahead of maintainability, cost of ownership, vendor reputation, ease of use and scalability.

4. For data management solutions, buyers are far and above interested in the performance and maintainability of these solutions. For data integration solutions, performance and functionality were top dogs with all others, except ease of use (second), distant thirds.

5. For BI solutions specifically, it seems that ease of integration is the runaway winner. Perhaps this explains the acquisition fever in our field these days with vendors scrambling to make their suite of products eventually fully integrated. Cost of ownership and performance were nearby seconds with ease of use, functionality, maintainability, scalability, and vendor reputation running a distant third.

Interesting, don’t you think? All the noise we hear about BI functionality, maintainability, and even scalability don't appear to be as important to the folks buying these products as the vendors think they are. Looks like Evalubase may serve a very important purpose in getting to the real requirements of our marketplace.

If you are interested in putting your two cents worth into their survey, all you need to do is register at their site.

As always -- Yours in BI success,


Posted April 13, 2005 8:21 AM
Permalink | 1 Comment |

I read an interesting article in the Business Intelligence Pipeline Newsletter recently asking which was the more difficult challenge - assuring data quality or integrating data from across your organization. They have a voting booth set up so you can cast your vote for which you believe is the more difficult task. I have my own opinion as well.

I voted for assuring data quality and, at the time of my vote, it appeared that the majority of voters agreed with me. Why? In my opinion, it is because of the assuring part of the task.

Data integration seems to be a much more straightforward task with more mature technologies, methodology, and practical expertise in the data integrators. Even the definition of data integration seems to be cut and dried. (Not always but at least you have a solid standard to go from -- a single version of the truth...)

I think we are still feeling our way through what it means to assure data quality. While there certainly is useful technology to help with data quality, so much of the assurance part is still heavily dependent on the human being (in this case, usually a business person)eyeballing the cleaned up data to verify its "quality". There don't seem to be very clear, standard methodologies or processes to follow either. And what are the metrics of quality? When to we reach a state of "quality"? And what exactly does quality data even mean?

Without answers to these fundamental questions, it seems to me that we will continue to struggle with this challenge more so than with that faced by data integrators.

Your thoughts?

Yours in BI success,


Posted April 11, 2005 1:03 PM
Permalink | 2 Comments |

Picture it -- San Diego, 70+ degree days, balmy nights, and some of the best BI discussions around. That was DCI's Business Intelligence conference this week. I was fortunate to be invited to speak and also came away with a few gems.

The conference was a three day event starting on Tuesday and covered a wide array of BI topics for the newbie team member as well as the experienced practitioner. There was even a light-hearted keynote by a plane jumper (parachute and all...). Jonathan Wu of Knightsbridge (feel better, Jonathan) and Barbara Gavin of DCI put together a solid line up including Bill Inmon, who spoke on the need for and difficulties with unstructured data, Colin White, speaking about the need to understand right time as opposed to real time data needs, and many others.

My own talk was in the last time slot on Thursday - clean up batter! I figured the audience would be cooked by then but they rallied around and we had a great discussion on the need to incorporate BI applications into the overall workflow of the business. Thanks to all of you that hung in there with me.

Hopefully I will see you at the next conference.

Yours in BI success!


Posted April 8, 2005 4:59 PM
Permalink | 1 Comment |

I just read an interesting article in April's CEO Magazine entitled "Fixing America's Future". Apparently, our public school systems have caught the attention of concerned CEOs. More and more of them are committed to championing school reform. Why? Read on to find out.

Perhaps Bill Gates expresses their concerns best. The Microsoft cofounder said in a February speech, "When I compare our high schools with what I see when I'm traveling abroad, I am terrified for our work force of tomorrow."

Louis Gerstner (retired IBM CEO), Craig Barrett (CEO of Intel), Jack Welch (retired GE CEO), Chris Gabrielli (founder of GMIS) Reed Hastings (founder of Netflix), Jeanette Wagner (former Vice Chairman of Estee Lauder), and numerous other CEOS all say similar things. The reason for their interest is that these executives face increasing difficulty in trying to fill job positions with recent US high school AND college graduates. They believe unless something is done now, the problem will only get worse.

These business professionals would like to apply the very same principles that made their companies some of the best in the world -- competition, accountability, and merit pay -- to public schools. While each has a different approach to solving this dilemma (some want charter schools or increased funding for schools, others want to extend school hours or more testing, many want to put real managers in the school districts and institute merit-based pay systems, etc.), they each recognize that there is no choice but to take direct steps now to improve the quality of our educational system.

I encourage you to read the article. Any country's future is dependent on its ability to innovate, create, and basically to think. Without a quality public education available to everyone, there is no doubt that this characteristic, so critical to success, will diminish.

As always, I welcome your comments.

Yours in BI success.


Posted April 6, 2005 4:22 PM
Permalink | No Comments |

I got an email recently from a friend of mine about an "evolving corporate office" dealing with a corporation's compliance - The Chief Compliance Officer (CCO). It is an interesting one that may have relevance to your enterprise. After reading the next sections, please add your own comments. Here is the rest of his email.

"Hi Claudia,

The organization chart in your recent B-eye article is missing an evolving corporate office that I think will play an increasing role in business intelligence. Part of the problem is that there is no commonly accepted initials for the CCO (Chief Compliance Officer). In some firms the title is Environmental Affairs, Chief Ethics Officers, Corporate Responsibility or Sustainability Officer. More prevalent in Europe, but you find Alcoa, DuPont and HP filling posts, as well as Starbucks and Green Mountain Coffee Roasters.

The significance is simply that the metrics and reporting associated with this issue is merging with SOX compliance, but right now the metrics work involved is not on the radar of the core business intelligence community. It's being developed by niche players, but I project a growing demand. The biggest reason for the demand isn't bleeding hearts, but tracking business benefits. Cutting waste drops heavily to the bottom line and more opportunities are being explored daily. So when firms like Herman Miller set goals for zero landfill by 2020 from their furniture operations, they generally have to pioneer the related business intelligence as well as the relevant business processes.

I think there is an opportunity to play a leadership role in the broadening beyond-compliance, business world."

As I said, an interesting and perhaps cntroversial position. What do YOU think? I look forward to hearing from you.

Yours in BI success,


Posted April 4, 2005 9:44 AM
Permalink | No Comments |

Three power houses, the Federal Trade Commission, the National Consumers League and Microsoft, have gone after alleged online identity theives -- those despicable people known as phishers. However, it appears that Microsoft is going one step further by filing 117 separate lawsuits against these creeps. I greatly approve of Microsoft's actions here.

What, you may ask, is phishing? It is those annoying and horribly misleading emails we all get in which the phisher tries to get you to give up personal information such as credit card and bank account numbers. The email directs you to a legitimate-looking website -- say for the Bank of America, complete with logos, corporate addresses, etc. -- and then asks you to fill in all your personal information.

According to Lydia Parnes of the FTC's Bureau of Consumer Protection, you have the power to stop these theives dead in their tracks -- by simply never responding to an email or pop-up asking for personal information. Just delete it. Sounds good but unfortunately 3 to 5% of the people getting these emails DO respond and give up some of their information. Gads! (For information on what you can do about identity theft, click here.)

The unforeseen consequence for our industry is that consumers will become even more reluctant to give any personal information to legitimate companies, seeking to be good CRM practitioners. This makes it even more difficult for these companies to better serve their customers through tailored service and product offerings.

Kudos to Microsoft and the others for trying to stop this scourge. By issuing the 117 lawsuits against these unidentifiable frauds (the defendants were all listed as John Doe), Microsoft hopes to "establish connections between the phishing scams worldwide and uncover the largest-volume operators", says Aaron Kornblum, a Microsoft attorney.

And good luck to those companies trying to get a better handle on their customers' needs and wants. While certainly a difficult thing to accomplish in today's environment, it will give your company bankable advantages over your competitors. Just don't abuse the knowledge you garner about your customers or you may find yourselves in a legal doghouse too.

Yours in BI success,


Posted April 1, 2005 10:06 AM
Permalink | No Comments |