Blog: Ronald Damhof Subscribe to this blog's RSS feed!

Ronald Damhof

I have been a BI/DW practitioner for more than 15 years. In the last few years, I have become increasingly annoyed - even frustrated - by the lack of (scientific) rigor in the field of data warehousing and business intelligence. It is not uncommon for the knowledge worker to be disillusioned by the promise of business intelligence and data warehousing because vendors and consulting organizations create their "own" frameworks, definitions, super-duper tools etc.

What the field needs is more connectedness (grounding and objectivity) to the scientific community. The scientific community needs to realize the importance of increasing their level of relevance to the practice of technology.

For the next few years, I have decided to attempt to build a solid bridge between science and technology practitioners. As a dissertation student at the University of Groningen in the Netherlands, I hope to discover ways to accomplish this. With this blog I hope to share some of the things I learn in my search and begin discussions on this topic within the international community.

Your feedback is important to me. Please let me know what you think. My email address is Ronald.damhof@prudenza.nl.

About the author >

Ronald Damhof is an information management practitioner with more than 15 years of international experience in the field.

His areas of focus include:

  1. Data management, including data quality, data governance and data warehousing;
  2. Enterprise architectural principles;
  3. Exploiting data to its maximum potential for decision support.
Ronald is an Information Quality Certified Professional (International Association for Information and Data Quality one of the first 20 to pass this prestigious exam), Certified Data Vault Grandmaster (only person in the world to have this level of certification), and a Certified Scrum Master. He is a strong advocate of agile and lean principles and practices (e.g., Scrum). You can reach him at +31 6 269 671 84, through his website at http://www.prudenza.nl/ or via email at ronald.damhof@prudenza.nl.

I have just read a very intriquing paper called 'A Common Approach for OLTP and OLAP using an In-memory Column Database', written by Hasso Plattner. 

It's not a revolutionairy new technical approach for Data Warehousing a Business Intelligence. It's just a series of smaller (mostly technical and some are even quite old) innovations that together could lead to a paradigma shift [1] in the area of Data Warehousing.

This paper is focussing on the transactional world, because that's where the disruption will originate. In short;

  • Ever increasing multi-CPU cores
  • Growth of main memory
  • Column databases for transactions (!)
  • Shared Nothing approach
  • In-memory access to actual data - historic data on slower devices (or not)
  • Zero-update strategies in OLTP (recognizing the imporance of history as well as the importance of parallelism)
  • Not in the paper; but I see datamodels for newly build OLTP systems increasingly resembling the datamodels of the HUB in the data warehouse architecture. 

Modern day Data Warehouses and Business Intelligence architectures incorporates all the above mentioned technologies/methods (well, they should!) and such an architecture therefor compensates for the weaknesses that is intrinsic for OLTP regarding OLAP. 

This paper is acknowledging the above technologies/methods and uses them in a OLTP context. The amazing thing is that the OLTP system is getting a lot faster and entail a lot less system maintenance (no indices, no aggregates, materialized views or what so ever, huge compression factors, etc..). 

BUT, maybe more interesting. What's the use of a data warehouse if the OLTP world is adopting these technologies/methods? Well, the case for a data warehouse becomes thinner. At least; data warehouses as we know it; ..a materialized store of (history and actual) data loaded from various sources ...

Simply put; with the above mix of technologies and methods we are able to stop propagating data. Or in other words; we can just leave the data where it initially is created. The data warehouse will then focus on the metadata (becomes hugely important!), business rules part (although advances in this area are also big), the integration part and the fit-to-task part (make it suitable for analytics, reporting, risc management etc..). Oh....data latency is non-existent.


Data architecture  - truly independent of it's task (whether it's transactional or informational). Could I live to see that? 

I advise people to read the paper from Lyytnen as well [1]:

Architectural innovations stand out as creative acts of adapting and applying latent technologies or potential to previously unarticulated user needs (Abernathy and Clark 1985). They radically deviate from an established trajectory of performance improvement, or redefine what performance means in a given industry (Chistensen and Bower 1996). They are radical (Zaltman et al. 1977) in that they significantly depart from existing alternatives and are shaped by novel, cognitive frames that need to be deployed to make sense of the innovation (Bijker 1987). Consequently, disruptive innovations are truly transformative (Abernathy and Clark 1985). To become widely adopted, disruptive architectural innovations demand provisioning of complementary assets in the form of additional innovations that make the original innovation useful over its diffusion trajectory (Abernathy and Clark 1985;Teece 1986). By doing so, disruptive innovations destroy existing competencies (Schumpeter 1934) and break down existing rules of competition.

I believe for the industry of data warehousing the above might apply. Nowadays, the new technologies and methods mentioned are increasingly used in the Data Warehouse and Business Intelligence scene. When it hits the OLTP scene it will radically change Data Warehousing and Business Intelligence as we know it.

How long will it take? Well, the latter alinea of the above quotation from lyytinen might slow real absorption down considerably:

To become widely adopted, disruptive architectural innovations demand provisioning of complementary assets in the form of additional innovations that make the original innovation useful over its diffusion trajectory (Abernathy and Clark 1985;Teece 1986). By doing so, disruptive innovations destroy existing competencies (Schumpeter 1934) and break down existing rules of competition.

SAP, Oracle and all other vendors of OLTP applications will have some work cut out for them. But I know that these quys are working hard......just listen to the SAP folks on their last summit....


[1] The disruptive nature of information technology Innovations - Lyytinen, Rose, 2003, MISQ 


Posted December 12, 2009 4:13 AM
Permalink | 47 Comments |

47 Comments

I thought it was going to be some boring old post, but it really compensated for my time. I will post a link to this page on my blog. I am sure my visitors will find that very useful.

This is such a great resource that you are providing and you give it away for free. I enjoy seeing websites that understand the value of providing a prime resource for free. I truly loved reading your post. Thanks!

I wanted to thank you for this excellent read!! I definitely loved every little bit of it. I have you bookmarked your site to check out the latest stuff you post.

A thoughtful insight and ideas I will use on my blog. You've obviously spent some time on this. Well done!

Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I'll be subscribing to your feed and I hope you post again soon.

I just couldnt leave your website before saying that I really enjoyed the quality information you offer to your visitors... Will be back often to check up on new stuff you post!

I just couldnt leave your website before saying that I really enjoyed the quality information you offer to your visitors... Will be back often to check up on new stuff you post!

It appears that you have placed a lot of effort into your article and I require more of these on the net these days. I sincerely got a kick out of your post. I don't really have much to say in response, I only wanted to comment to reply wonderful work.

I just couldnt leave your website before saying that I really enjoyed the quality information you offer to your visitors... Will be back often to check up on new stuff you post!

Aw, this was a really quality post. In theory I'd like to write like this too - taking time and real effort to make a good article... but what can I say... I procrastinate alot and never seem to get something done.

This is such a great resource that you are providing and you give it away for free. I enjoy seeing websites that understand the value of providing a prime resource for free. I truly loved reading your post. Thanks!

Aw, this was a really quality post. In theory I'd like to write like this too - taking time and real effort to make a good article... but what can I say... I procrastinate alot and never seem to get something done.

Aw, this was a really quality post. In theory I'd like to write like this too - taking time and real effort to make a good article... but what can I say... I procrastinate alot and never seem to get something done.

I can see that you are an expert at your field! I am launching a website soon, and your information will be very useful for me.. Thanks for all your help and wishing you all the success in your business.

It appears that you have placed a lot of effort into your article and I require more of these on the net these days. I sincerely got a kick out of your post. I don't really have much to say in response, I only wanted to comment to reply wonderful work.

It appears that you have placed a lot of effort into your article and I require more of these on the net these days. I sincerely got a kick out of your post. I don't really have much to say in response, I only wanted to comment to reply wonderful work.

We just couldnt leave your website before saying that we really enjoyed the quality information you offer to your visitors... Will be back often to check up on new stuff you post!

Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with more information? It is extremely helpful for me.

I am really enjoying reading your well written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!

We are a group of volunteers and starting a new initiative in a community. Your blog provided us valuable information to work on.You have done a marvellous job!

I wanted to thank you for this excellent read!! I definitely loved every little bit of it. I have you bookmarked your site to check out the latest stuff you post.

It appears that you have placed a lot of effort into your article and I require more of these on the net these days. I sincerely got a kick out of your post. I don't really have much to say in response, I only wanted to comment to reply wonderful work.

Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I'll be subscribing to your feed and I hope you post again soon.

Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with more information? It is extremely helpful for me.

Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I'll be subscribing to your feed and I hope you post again soon.

Hi buddy, your blog's design is simple and clean and i like it. Your blog posts are superb. Please keep them coming. Greets!!!

Hi buddy, your blog's design is simple and clean and i like it. Your blog posts are superb. Please keep them coming. Greets!!!

I know this is really boring and you are skipping to the next comment, but I just wanted to throw you a big thanks - you cleared up some things for me!

This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here! keep up the good work.

I just couldnt leave your website before saying that I really enjoyed the quality information you offer to your visitors... Will be back often to check up on new stuff you post!

I can see that you are an expert at your field! I am launching a website soon, and your information will be very useful for me.. Thanks for all your help and wishing you all the success in your business.

I know this is really boring and you are skipping to the next comment, but I just wanted to throw you a big thanks - you cleared up some things for me!

It appears that you have placed a lot of effort into your article and I require more of these on the net these days. I sincerely got a kick out of your post. I don't really have much to say in response, I only wanted to comment to reply wonderful work.

I know this is really boring and you are skipping to the next comment, but I just wanted to throw you a big thanks - you cleared up some things for me!

I know this is really boring and you are skipping to the next comment, but I just wanted to throw you a big thanks - you cleared up some things for me!

Hi buddy, your blog's design is simple and clean and i like it. Your blog posts are superb. Please keep them coming. Greets!!!

It appears that you have placed a lot of effort into your article and I require more of these on the net these days. I sincerely got a kick out of your post. I don't really have much to say in response, I only wanted to comment to reply wonderful work.

This is such a great resource that you are providing and you give it away for free. I enjoy seeing websites that understand the value of providing a prime resource for free. I truly loved reading your post. Thanks!

It appears that you have placed a lot of effort into your article and I require more of these on the net these days. I sincerely got a kick out of your post. I don't really have much to say in response, I only wanted to comment to reply wonderful work.

Cool, there are actually some great points on here some of my readers may find this worthwhile, will send a link, thanks.

This helps me a lot, thanks for the information.

Aw, this was a really quality post. In theory I'd like to write like this too - taking time and real effort to make a good article... but what can I say... I procrastinate alot and never seem to get something done.

You made some good points there. I did a search on the topic and found most people will agree with your blog.

I know this is really boring and you are skipping to the next comment, but I just wanted to throw you a big thanks - you cleared up some things for me!

Houses are quite expensive and not everybody is able to buy it. But, mortgage loans was invented to support people in such hard situations.

I wanted to thank you for this excellent read!! I definitely loved every little bit of it. I have you bookmarked your site to check out the latest stuff you post.

I just linked this article on my facebook account. Its an interesting read for everyone.

Leave a comment