Blog: Ronald Damhof Subscribe to this blog's RSS feed!

Ronald Damhof

I have been a BI/DW practitioner for more than 15 years. In the last few years, I have become increasingly annoyed - even frustrated - by the lack of (scientific) rigor in the field of data warehousing and business intelligence. It is not uncommon for the knowledge worker to be disillusioned by the promise of business intelligence and data warehousing because vendors and consulting organizations create their "own" frameworks, definitions, super-duper tools etc.

What the field needs is more connectedness (grounding and objectivity) to the scientific community. The scientific community needs to realize the importance of increasing their level of relevance to the practice of technology.

For the next few years, I have decided to attempt to build a solid bridge between science and technology practitioners. As a dissertation student at the University of Groningen in the Netherlands, I hope to discover ways to accomplish this. With this blog I hope to share some of the things I learn in my search and begin discussions on this topic within the international community.

Your feedback is important to me. Please let me know what you think. My email address is Ronald.damhof@prudenza.nl.

About the author >

Ronald Damhof is an information management practitioner with more than 15 years of international experience in the field.

His areas of focus include:

  1. Data management, including data quality, data governance and data warehousing;
  2. Enterprise architectural principles;
  3. Exploiting data to its maximum potential for decision support.
Ronald is an Information Quality Certified Professional (International Association for Information and Data Quality one of the first 20 to pass this prestigious exam), Certified Data Vault Grandmaster (only person in the world to have this level of certification), and a Certified Scrum Master. He is a strong advocate of agile and lean principles and practices (e.g., Scrum). You can reach him at +31 6 269 671 84, through his website at http://www.prudenza.nl/ or via email at ronald.damhof@prudenza.nl.

I often critisize vendors and others for not being thorough enough. Now it's time to critisize science a bit as well as those consultants and analysts that are 'riding the wave' of negative sentiment surrounding data warehousing.

In several papers I am reading at the moment (ranging from MIS Quarterly, Information & Management, Decision Support Systems and many more journals) I encounter something similar.

Please read the following quotes:

"....the road to DW success has been littered with failures [43,63,80]"
"....nearly half of all DW initiatives end up as failures [38]"
"According to a press release 2005 by Gartner: through 2007 more then 50% of data warehouse projects will have limited acceptance, or will be outright failures."

The above quotes I got from one paper in one paragraph in Decision Support Systems (paper is from 2008 - very recent), which is an important journal in our field of expertise. Since I encounter these statements over and over again, I decided to follow up on the citations.

Let me begin by dismissing the quote refering to Gartner's press release. That's not really a sound scientific basis.....So we are left with two more quotes.

[43] C.Hall Corporate use of data warehousing and Enteprise Analytic Technologies, Arlington, Massachusets, 2003 URL: www.cutter.com
Ronald: can not validate the information. You need to buy a report......

[63] S.Kotler When enterprise hit open road: move beyond the silos and let the idead roll, Teradata Magazine 3 (3) 2003
Ronald: I read this article...now it becomes shocking. Let me quote this article:

"According to a recent article in Information Week, "41% of all companies surveyed by the Cutter Consortium, an IT consultant and market analysis firm, have failed data warehouse projects, and only 15% call their data warehousing efforts to date a major success."

Ok....this citation is actually referencing the first one [43] (I think...can't access it). We got a loop here....

[80] M.Quaddus, A.Intrapairot, Management Policies and the diffusion of DWH: a case study using dynamic based decision support systems, Decision Support Systems 31 (2) 2001, 223-240
Ronald: ok, this is becoming complex. I quote this paper:

"..quite a few DW projects end up in failure even before full implementation owing to lack of immediate substantial economic returns on massive investment [24,25,67]"

So a citation is used in a paper that refers to a claim in another paper but with different citations (you guys still with me here?). So we end up with three more citations which are off course even further in time, let's examine these:

[24] R. Hackathorn, Data warehousing energises your enterprise, Datamation 41 (2) 1995. 38-42
- I could not find this article, so I was not able to validate the claim that was being made. It sounds to me like some sort of column though. But again - can not validate.

[25] C. Horrock, Making the Warehouse Work, 1996, available from http://www.computerworld.comrsear . . . -htm1r9606r960624DW1SL96dw10.html.
- I could not find this article, so I was not able to validate the claim that was being made.

[67] The Siam Commercial Bank's Staff, Data Warehouse Questionnaire and Interview, (6 January-28 February 1998). Personal Communication.
- I could not find this 'article', so I was not able to validate the claim that was being made.

[38] L.Greenfield, The Data Warehousing Information Center www.dwinfocenter.org, december 19, 2003
Ronald: This is a link to a whole site.......oh my, how on earth can you make a reference to a whole site??


To summarize things; I was not able to establish any (empirical) evidence that would support the claim made in the paper of DSS (and in several other papers as well) - which is by the way quite a recent paper (2008). Somehow we are being made to believe that Data Warehousing are failure prone. Increasingly I encounter consultants and analysts that fuel this negative sentiment surrounding data warehousing. I challenge anyone to deliver some real (empirical) evidence. As for now - I suggest we all use caution in communicating that Data Warehouse undertakings tend to fail a lot.

On the subject, there is one interesting piece of paper from TDWI, written by Hugh J.Watson (I think in 2006) that seems to be relevant and hitting the nail on the head by saying - and I quote:

"The data suggests that whether data warehouses are failure-prone depends on one's definition of "failure." Varying with the architecture implemented, there is approximately a 32-47 percent chance that a warehouse will be behind schedule, and a 30-43 percent chance that it will be over budget. However, this does not mean that the warehouse will not succeed. By a more global measure of success, only 10-20 percent of warehouses are potentially in trouble, while the others are either up-and-coming systems or runaway successes"

Ronald: And yes, Watson is using an empirical basis.

Posted June 22, 2009 3:24 AM
Permalink | 1 Comment |

1 Comment

I thought it was going to be some boring old post, but it really compensated for my time. I will post a link to this page on my blog. I am sure my visitors will find that very useful.

Leave a comment