Every woman has a moment where she thinks to herself—or shrieks aloud, depending on the circumstance—“Omigod. I’m turning into my mother.” I had that moment last Thursday. And, to make matters worse (or better), I wasn’t standing before a mirror or admonishing a child, I was meeting with a client.

You see my mother had a parenting style that was ahead of its time. She was never pedantic. She didn’t lecture or finger-wag. She raised neither voice nor paddle. She simply encouraged us to think about the “eventual outcome” of our decisions and weigh the consequences of our actions. She was sober and deliberate and encouraging. There were plenty of skinned knees.

So here I am sitting the headquarters of this consumer packaged goods firm talking to a director. He wonders why he has five full-time people managing the company’s product item master. The company runs a packaged ERP system that requires a heavy amount of customization and lots of manual effort. The director knows that there are business processes that aren’t automated. He also knows that the system’s data is bad—and getting worse.

For instance, every time a new part appears on a purchase order, these five busy managers schedule a meeting and discuss the new product. They try to determine whether the new product resembles another product, if it matches a current product, if it’s already been defined or could otherwise already exist on the item master list. Sometimes the product has more than one manufacturer—think 3/4-inch Number 5 bolt, ten threads-per-inch coming from multiple factories—and they have to work that out too. Most of the time the team suspects the product already exists in the database. But confirming this isn’t as easy as simply creating a new record for the product. Sometimes these meetings get tense. Business people have to be called, and sometimes business executives show up to mediate. And so it goes.

I found myself asking the director what his desired outcome was. I inquired whether the manual management and reconciliation of data was worth the effort. I speculated about other work the five managers could be doing if they weren’t tussling over data. I asked the director to think about why he was unwilling to front the budget money for a data quality tool that could profile the data and automate its reconciliation. I was sober and, other than the welfare of my client, I had no agenda. Nothing doctrinaire, just some gentle guidance.

That’s when my Mother Moment hit me. But, in a supernatural moment of kismet, it’s also when my client saw the light. “Well, could we work up some sort of prototype project?” he asked tentatively. Breakthrough. Thanks, Mom!

Technorati tags: data quality, data quality tool, data reconciliation

I’ve never written about this before. In the late 1980s, I worked in Germany on a project with the goal of tracking terrorists.

Back then I had both the physical and intellectual stamina required of a system engineer for a start-up database vendor, often dispatched to far-flung locales with a day’s notice to help make a database go faster or help a customer with some new product feature. When my boss told me to be in Wiesbaden on Thursday to “help a new customer integrate its data models,” I bought a book on the Rhein-Main region, studied up on the vernacular, and booked my ticket.

The Bundeskriminalamt is German amalgam of the FBI and the CIA. The bright, airy layout of its Wiesbaden headquarters belied the serious work that went on inside. In an architectural flourish that in retrospect was counterintuitive, the BundesKriminalAmt building was walled with windows. The outside of the building was surrounded by glass so expansive that black bird decals were pasted across it in intervals so the crows wouldn’t fly into the structure and knock themselves unconscious, or worse.

I remember entering the massive building every day, enduring the same line of questioning and briefcase search from stern security personnel, and handing over my U.S. passport, which I retrieved promptly at six o’clock before the shift change.

I was there to integrate disparate database designs, some hierarchical, some flat files, none of them matching, into a single relational model. It wasn’t the standard bill-of-materials project or even one of the customer models we’d recently begun working on for our clients. This was a terrorist database, one intended to interconnect the various police stations throughout Germany via a network of common information about terrorists. The data involved border crossings, Visa numbers, and aliases. Back in the days of manual data profiling, we discovered data elements like “S.A.S” and “Kalashnikov.”

My project was explained to me with the matter-of-fact demeanor associated with German efficiency. I worked with a team of focused professionals, all of us charged with deconstructing the various files from the disparate police stations and building a common model. We worked hard and deliberately, in the German fashion. Our boss, Herr Lietz, a lifelong civil servant, occasionally invited me home for dinner with his family, all of whom politely ate in the Continental style, knife in the right hand, fork on the left. Herr Lietz was gracious and smart. He kept his secrets to himself.

After work I’d sit on my bed at the Hotel Schwarzer Bock idly watching German talk shows and mulling over the information we were gathering. I’d wake up early and have a prototypical breakfast of fruit, muesli, and quark (an enduring love my hips resent). On weekends I would take the train to Baden-Baden or visit friends in Strasbourg, where we would dine in the shadow of the cathedral and debate our differences.

On reflection, I was hyper-intent on the job at hand but oblivious to its foreshadowing. None of us could have imagined that Germany would be the incubator of an organized plot against the West, or that terrorist cells in Hamburg would be directly linked to a tragedy on the scale of 9/11. I was laying the foundation for new information, proselytizing my belief in the relational model, building a data dictionary from scratch. I was representing my company, delivering then-new data warehouse technology that would make the information so much easier for non-technicians to access and understand. I was determined to complete attribute lists, represent roles, and avoid dangling foreign keys. I was twenty six years old.

Technorati tags: CDI, data reconciliation, data modeling