I have been a BI/DW practitioner for more than 15 years. In the last few years, I have become increasingly annoyed - even frustrated - by the lack of (scientific) rigor in the field of data warehousing and business intelligence. It is not uncommon for the knowledge worker to be disillusioned by the promise of business intelligence and data warehousing because vendors and consulting organizations create their "own" frameworks, definitions, super-duper tools etc.
What the field needs is more connectedness (grounding and objectivity) to the scientific community. The scientific community needs to realize the importance of increasing their level of relevance to the practice of technology.
For the next few years, I have decided to attempt to build a solid bridge between science and technology practitioners. As a dissertation student at the University of Groningen in the Netherlands, I hope to discover ways to accomplish this. With this blog I hope to share some of the things I learn in my search and begin discussions on this topic within the international community.
Your feedback is important to me. Please let me know what you think. My email address is Ronald.firstname.lastname@example.org.
Ronald Damhof is an information management practitioner with more than 15 years of international experience in the field.
His areas of focus include:
Business Intelligence vendors seem to embrace collaboration (I am still struggling whether this software is any different from the groupware we had in the 90's) . As an example please take a look at SAP streamwork at youtube. I am gonna be blunt here; this type of software is completely useless, unless the organization is willing to fundamentally change its decision making process.
Let me try to make my point here with the help of giants like Galbraith, Daft, Davenport and some..
There are basically two information contingencies; Uncertainty and Equivocality.
What we all knew was true, but could not get across to management, is now more scientifically proven. The decision process regarding the outsourcing of a DSS is influenced by significantly other characteristics, when compared to OLTP. If you are interested in the details, the theory and the underlying data, please read:
considered when outsourcing an IS system:an empirical examination of
the impacts of organizations size, Strategy and the object of a
decision (DSS or OLTP).
B.Berg and A.Stylianou in the European Journal of Information systems (2009 18, 235-248)
I still encounter organizations who are stuck in the OLTP world, even when the object of decision regarding outsourcing is completely different on many dimensions. They tend to use the same decision process regarding the outsourcing as they always did...whether they outsource an ERP, a CRM system a data warehouse or a more elaborate BI system.
Lately I come across vendors of OLTP systems that engage highly in tied selling. Now - if there is complete transparency to the customer- this is off course no problem. But, the sad thing is, this transparency is usually extremely bad or even non existent. This tied selling combined with (complete) lack of transparency has caused severe damage to the economy as we have seen especially in the financial sector. In Europe the European Commission is highly active in forcing vendors to stop tied selling (for example Internet Explorer and the OS of Microsoft). Vendors receive huge fines if they engage in these practices. Unfortunately these practices are also quite mainstream in the data exploitation industry.
Lately I come across vendors of OLTP systems that engage highly in tied selling. Now - if there is complete transparency to the customer- this is off course no problem. But, the sad thing is, this transparency is usually extremely bad or even non existent.
This tied selling combined with (complete) lack of transparency has caused severe damage to the economy as we have seen especially in the financial sector. In Europe the European Commission is highly active in forcing vendors to stop tied selling (for example Internet Explorer and the OS of Microsoft). Vendors receive huge fines if they engage in these practices.
Unfortunately these practices are also quite mainstream in the data exploitation industry.
Yep - bringing up an oldie, but still o so true.
Financial assets: A single Euro can only be spend once.....
Human captial: They can just walk out of the door and leave or be hunted by your competitor
Not so with your data; It is uniquely yours, closely connected to your business language, full of nuance, in the proper context and the ability to procreate new data. Every organization can use it's data differently to create it's own niche. When data is used by anyone in your organization, it will not be consumed and it will not expire (a patent will). There is simply no other asset that anyone can think one that owns these capabilities.
To make the case even stronger; the movement of data in your organization is your lifeblood. Any company will simply seize to exist if this data is of bad quality. Orders will not get through, customers credit information can not be seen and invoices can not be send.
Another fascinating attribute of data is that it defines the other assets; financial data, property data, employee data. Badly managed data will jeopardize the management of the other assets big time. So, it's kind of a 'super', ultimate asset.
Nicholas Carr wrote in 2003 in the Harvard Business Review 'IT doesn't matter'. Systems, infrastructure, services, hardware, software, ERP...whatever...they will and probably are already commodities. They are not a differentiating factor in competitive advantage, sustainability or reliability. Data is however always unique, competitors can not get hold of it (well....unless you got a security problem) and if they got it, it will probably be hard to use, since data is often very closely linked to the organizational lingua franca.
Data must be nurtured, managed aggressively and - most of all- must be enabled to be exploited to the max in order to get a good return.It is by far the ultimate proprietary asset.
I believe that in business nowadays huge amounts of money are spend on very sophisticated technology, financial controls and process management (the process taliban). The marginal returns from ever increasing investments into these developments will be small. Investments in data are quite the opposite. The opportunities are limitless. And, more importantly, any investment you make is proprietary to your organization. This is the true definition of competitive advantage.
This posted is inspired and sometimes quoted by T.Redman and his book 'Data Driven'
July 2007 I had a first aquantaince with Dan Linstedt - via LinkedIn.
I worked for a large government organisation as the architect for Data
Warehousing and Business Intelligence. I seriously considered Data
Vault as the standard methodology for the Enterprise Data Warehouse. I
worked for over 2 years for this govenment organisation and my choice
for the Data Vault is still a very good one.
November 2007 I got
Dan Linstedt to the Netherlands and got the first 27 peeps trained and
certified of that same government organisation. I had dozens of
discussions with architects, management, developers, etc...but in the
end the consensus on choosing Data Vault was huge.
2007 Data Vault sky rocketed in the Netherlands. Together with the
Genesee Academy and DNV I think we certified over 200 people and I dare
to say that most new EDW projects nowadays in the Netherlands have
chosen Data Vault as the prime methodology.
Before I get
criticised for sanctifying one methodology despite the various (non)
functional requirements; I do believe that ANY architecture needs to be
in line with the business-, application- and technical architecture. So
choosing one methodology on forehand is just plain wrong. There has to
be sound justification and alignment for any choice in your
architecture. Whether it's a tool, a methodology or whatever.
2 years we (Genesee Academy, me, DNV) certified over 200 people - most
of them extremely seasoned and respected experts in our field. On
average these consultants are very critical on any new 'trick in the
book'. I dare to say that most of the experts that were certified were
convinced that Data Vault has got extreme value and potential. The
proof of the pudding is in the eating; Data Vault got many
implementations in the Netherlands the last two years and it's growing
I am confident that Data Vault in the years to come will hit Europe as hard as it has hit the Netherlands.
I wonder......I and am still baffled by it. The attention for Data
Vault in the United States is extremely limited, why is that? Even a
respected institute as TWDI is not paying much attention. Why? Experts
in the field do not mention Data Vault what so ever. I just don't get
it. Can anybody enlighten me?
For those of you not familiar with Data Vault --> read my articles.
In the 'Neverland' of decision support, Business Intelligence has completely taken over all attractions.
Let's name a few; BI, Neuro BI, Ambient BI, Sentient BI, Process driven BI, process Intelligence, Pervasive BI, BI 2.0, BI as a Service, BI for SOA, SOA for BI, Mobile BI, google BI, Personal BI, BI-Tools-where-you-do-not-need-a-DWH-and-ETL-Buy-it-now, Mashups-the-new-BI-Desktop (oh my god), BI-in-the-box,Business Analytics,BI-in-the-cloud (love this one!), BI-virtualization, Agile-BI,Operational-BI,Decision Intelligence, Decision Management, Even driven analytics, Complex Event Processing, BAM, Collabarative Decision Making......
Do not get me wrong; I certainly do not dismiss them all.....just most of them :)
Most of these 'attractions' are highly sponsored, look super-dooper on the outside and if you won't sit in them you will loose out big time (so they say). However, when you finally decide to ride them you feel that they are not that stable, not really safe and there is hardly any enjoyment when you exit them (but you can tell you neighbour you dared to ride it!). To put it in other words, not really grounded in theory and the relevance for practice is extermely hard to find.
There are however some attractions in this Decision Support 'Neverland' that are very much dusty, spiderwebs all over the place, but the attraction is still extremely solid and if we would overhaul it with new architectural insights and technology, it could be a smash-hit. These attractions are named DSS, EIS, ESS....
I feel that we - as an industry - failed miserably in continuing on the path that was made for us by people like John Dearden, John Rockart, David Delong, Ralph Sprague, Hugh Watson, Steven Alter, Daft and Lengel, Peter Keen, Michael Scott Morton, Herbert Simon, Henry Mintzberg and many more.
Dan Power is one of those brave souls who is standing with the ticketbox - selling tickets for his 'attraction'. I recommend people to read his last written article as well as the blog post written by Wouter van Aerle.
Finally, I wanna contribute to the Decision Support 'Neverland': BI goes Retro
In 1986 Daft and Lengel published an article in Management Science:
ORGANIZATIONAL INFORMATION REQUIREMENTS, MEDIA RICHNESS AND STRUCTURAL DESIGN.
A very interesting article that combines the processing of information by decision makers, the use of structural mechanisms and organizational design. The latter one - organizational design - I will not discuss thoroughly in this blog. Just remember that the choice of structural mechanism to overcome uncertainty/equivocality will impact the organizational design.