The kinds of charges that appeared were interesting: $33.10, $45.00, and $70.00. Of the 3 charges, only one was not a round-dollar amount, and the second and third charges were done at the same location almost at the same time.

Back in April, I wrote an article for B-EYE-Network on the use of Benford's Law for Information Analysis. In this article, I described a digital analysis phenomenon regarding the distribution of numeric digits in large data sets. The truth is, Benford Analysis has been used primarily as an auditing technique to look for fraudulent behavior , and I am confident that (with a little thought) a reasonable use of the technique could help in identifying transaction patterns involving different duplicated credit cards.

Individuals are likely to repeat their bad behavior, and even if they think that they are creating random dollar amount charge sequences, each may reflect a particular signature that identifies the perpetrator, and by analyzing the geographic density of the illicit charge locations, pinpoint a reasonable location to start to track down the offenders.

Anyone out there with experience in fraud detection using Benford's Law? What do you guys think?

On the one hand, a number of data quality experts advocate ensuring that the data is clean when it enters your system, which would support the decision to cleanse the data at the intake location.

On the other hand, since not all names and addresses input to the system are used, cleansing them may turn out to be additional work that was not necessary. Instead, cleansing on use would limit your work to what is needed by the business process.

Here is a hybrid idea: cleanse the data to determine its standard form, but don't actually modify the input data. The reason is that a variation in a name or address provides extra knowledge about the individual - perhaps a nickname, or a variation in spelling that may occur in other situations. Reducing each occurrence of a variation into a single form removes knowledge associated with potential aliases, which ultimately reduces your global knowledge of the individual. But if you can determine that the input data is just a variation of one (or more) records that you already know, storing the entered version linked to its cleansed form will provide greater knowledge moving forward.

Every 10 years, the US Census Bureau conducts a census, and as part of that process, collects a huge amount of demographic data about on avery granular level, geographically. They then spend the next 5+ years analyzing the data and preparing it for release, while at the same time preparing for the next decennial census.

The problem is that sometimes, by the time the decennial data is released it no longer accurately reflects an area's demographics. For example, consider how rapidly real estate prices have risen in the past 5 years - yet 2005 home prices are not captured in Census 2000 data. Similarly, the Tiger/Line data that contains information about street addresses is occasionally updated, but new streets and subdivisions are constantly being built, so it is likely that there are omissions in the Census data set.

There are many other public domain, public records, or generally available data sets that are of great interest to the BI community. So here is the challenge: Tell me how you feel about a project to take on a publicly available data set and create an "Open Source" approach to managing various approaches to maintaining and presenting that data. One example might be taking the Census decennial data and formulating it into a relational data structure mapped across the geographic Tiger/Line data? Pose your ideas as comments to this enrty...

So who were the four people that triggered this though process? First, as I was exiting the DC Metro, I was passed by a person who reminded me of a childhood friend with whom I had briefly reestablished a connection a few years back when I tracked down his contact information via Google.

Second, I have been exchanging voice mails with my friend Greg Elin, who is a really bright guy. Greg and I worked together on a few projects, one involving a (now questionable) idea for a web-based service that archived banner ads.

Third, I got an email from a friend and data quality colleague who had been tasked with evolving a solution to a rather sticky (and most likely intractable) Customer Data Integration project.

Fourth, there is a mail list for the employees of the company where I worked for my first job, Compass (also known as Massachusetts Computer Associates). These emails suggested that we migrate the email list to Yahoo Groups, and there seems to be some concurrence to this.

But when you take a look at Yahoo's Privacy Policy, you will see that Yahoo collects information about the transactions of its members, and uses them for research, personalization, targeting, and aggregation. One can extrapolate and assume that Yahoo tracks the relationships and micro-communities associated with the individuals that subscribe (a weak link) or participate (a stronger link) within that group. Determining the strength of the various links, the "spheres of influence," and learning "who knows who" can add a significant amount of value to an ad targeting strategy.

For example, if a particular person clicks through a specific banner ad, one might assume that other individuals within that person's sphere of influence might also have a predilection to respond to a similar ad? Of course, this can be used to refine ad targeting, increase response rates, and consequently increase the charge for ad placement.

Okay, so what does this have to do with my walk down memory lane today? The fact is that I can qualify each of those relationships with different kinds of attributes, and those attributes (and their value sets, and magnitudes, etc.) might contribute even more to analyzing its BI value. More on this topic to follow...

In fact, only recently had there been any real movement in the Data Reference Model arena, and what had been released was an XML model intended to represent a way to register data sets for the purposes of data exchange. Now, considering that Mr. Daconta comes from the Department of Homeland Security, it is not surprising that the desire to effectively share information among government agencies is helping to drive the effort forward. One consideration, though, is that the rest of the FEA focuses more on assessing and documenting government investment in technology infrastructure and how that infrastructure is put to use to improve the way Federal agencies manage their IT investments. Because of this, my perception is that there is a bit of a disconnect between the rest of the FEA and the Data Reference Model part of it.

Luckily, our friends from the Federal Data Registry Users Group are pretty smart, and were able to direct some good questions to those defining the DRM, and Mike Daconta spent some time today addressing some of those questions. The conclusion was very inspiring, in that there may be a good opportunity to inject some good ideas and guidance into the use and value of metadata into a process that affects most, if not all, agencies in the Federal Government.

For more information on the continuing saga of the FEA DRM, see the project's Wiki.

Basically, the prospect already had a good grasp of data analytics; they already did some rudimentary analysis, segmenting their customer base into qualitative segments, and using predictive analysis about customer/product lifecycles that indicated continued customer loyalty or attrition. They were interested in doing more of this kind of exploration, yet their staff is working to capacity, so it is the president that actually does all the analysis.

The real question turned out to be: what would his company's return on investment were they to engage us to build an analytical platform? The impasse I saw was that I find it hard to project an ROI when I don't have much of an idea of what is buried within their data asset, nor can I predict what kinds of questions the organization needs answers, nor, most importantly, how the organization will make the answers actionable.

At the end of the conversation, we were both asking what the next steps would be. I proposed a longer conversation to determine business analysis criteria, and he demurred, citing a busy schedule, and a lack of being overwhelmed by the potential of what I was describing. In other words, because I could not project actionable knowledge with a calculable ROI before deploying the system, there was no business case to build a system.

This highlights a gap we often see in client BI engagements: the technology provider is building a capability that allows the client to ask the right questions, while the client expects the provider to provide the right answers. I believe that the most successful programs are the ones where expectations are "level-set": the providers and the clients team to determine how the right questions will improve the business, there is an understanding of how the technology enables asking the right questions, and the client has a plan to exploit actionable discoveries.