In Part 1 of this article, you learned about Data Projects and how they are different from Application Development Projects.  You followed along with my co-worker Alicia as she and I shared knowledge about our projects and especially the three Data Project Pitfalls I have encountered in my past projects.  Now you will find out how Alicia learned to avoid the Data Project Pitfalls and enhance the project lifecycle methodology she was already using with the Business Rules Approach in order to ensure a successful Data Project implementation.

The Business Rules Approach for a Data Project

When the approach is…	The results will be…
Define high-level business goals, strategies, and tactics	Clear description of business motivation and definition of importance
Define business terms and model business facts	Clear definition of scope and project boundaries
Discover and define data-related business rules	Business-recognizable foundation for logical data model
This approach is the Business Rules Approach You must show your business clients this really works!

I call this an approach because it does not replace a project methodology; it enhances the methodology for a Data Project.  I call this the Business Rules Approach because:

1. It focuses on the business, not the system, not the project, and not the technical solution.
2. It requires all types of rules to be documented at all levels of the project.
3. It requires that IT understand the business and the big picture.
4. It allows the business rules documentation and deliverables to become self-documenting test criteria.

The enhancements to the project methodology center around the project deliverables.  In the Data Project it's critical to understand business goals, strategies, and tactics … because each of these will identify the major terms, or subjects, that are of interest to the business.

In this approach you work down from the high level and work into the details by asking for definitions of business terms and by modeling business facts.  At each level of detail you discover, define, and document all the data-related business rules.

Alicia made a very insightful observation — since this process is very detailed and will consume time with the business subject matter experts who already think they know what they want and need, the Data Management team must demonstrate how this approach works and get acceptance in order to move forward.  It's going to be critical to show your business clients what this is all about and that it really works.  We were successful by using real-world examples of actual projects where we were successful in this approach … and also where we could have encountered the pitfalls.…

Case A — Strategic Business Reporting

We recently had a project where we were asked to replace a legacy reporting system that was used by two very strategic analytical groups.  The business had approached us with the "just give me everything" statement (Remember Pitfall #1?) since they were pretty much replacing what they already had.

However, we were able to show how using the business rules approach to their requirements would be worth the time and effort because, even though they weren't changing the source of their data and the data requirements were the same, the level of detail was changing, and this meant that their old system and old assumptions were no longer valid.  What they used to have would not work any longer, and they had to re-think everything about their requirements.

We used our business rules approach to the analysis of their requirements:

• Existing reports were analyzed for requirements and business rules.
• New business strategy was layered onto the existing strategy, to understand the impact of changes and level of detail.
• Business rules from the legacy reporting system were implemented in EDW views.
• Legacy reporting system and replacement reports were compared and reconciled.

As sound as the process is, we still uncovered many hidden and deprecated business rules that were missed in the delivery of the replacement universe, which even surprised the business in some cases and reinforced the importance of complete analysis of business goals and strategies.

Case B — Forecasting and Planning System

On this project, we had a very large effort to replace an existing forecasting and planning system.  Our goal was to provide the data needed from the EDW instead of having to pull data from all the various data sources into the system.  We were given an extensive list of data and told, Give us all of this and we'll take it from there.  (Remember Pitfall #2?)

However, by following the business rules approach, we were able to show:

• Some of the data elements on their list were not sourced the way that they expected.
• Some of the business rules were hidden in a secondary source or report or application screen.
• No one on the business team could articulate the business rule that created the requirement!

Our approach included a complete definition of every data element, and obtaining agreement and consensus from every business unit interested in the data.

Here's a humorous example — we were given a requirement to get a count of "Virgin Room Nights."  No one in the requirements session had ever heard of a "Virgin Room Night."  I could only guess at the definition of a "Virgin Room Night" and my imagination ran wild.  We asked for the definition, and we found that a "Virgin Room Night" is one night in one room that was sold to a Guest by a wholesaler by the name of Virgin Vacations, which is one of the companies affiliated with Virgin Airlines and is a travel partner to the Corporation.  In this particular example, there would have been no way, on the face of it, to deliver on a requirement for "Virgin Room Nights" without seeking out the complete definition.  The requirement, written with the definition in mind, is to count the number of rooms occupied on a given night by the wholesaler name.  To do this you have to know the wholesaler code, the wholesaler name, the service date, and the occupancy status.  It was never just one thing; it was really four or more things.

Every one of the 338 data elements/labels had to be analyzed and mapped in order to provide the interfaces to the application.

Case C — Customer Managed Relationships

Our marketing partners needed a more advanced system to perform direct marketing tactics and wanted to integrate data from their new customer relationship management system with all sorts of other legacy data, including reservations and stays.  We created our models and metadata, and documented our business rules and our audit processes to populate the database, as required.  When it came time to build our User Acceptance Test Plan, our marketing partners said that they would sign off when the data in the source could be reconciled and matched to the data in the new system.  When asked for acceptance criteria, they said that they would know it when they saw it.  (Remember Pitfall #3?)

Instead, we were able to take each one of our business rules and create a test scenario to test each rule, mapping, and integration point possible.  Some business rules were designed to improve Data Quality by rejecting outliers and "bad" data … so variances between the source system and the data warehouse were expected.  Using this approach, the business rules themselves became the basis for the acceptance criteria, and reconciliation could happen much more easily.

At this point, Alicia was looking forward to working on her first data project.  But she was hungry for more and wanted to know how all the other people involved in her data project were going to respond to this approach.  As well as the Business Rules Approach works for the business, it's even better for all the technical teams involved in supporting the data project.

Data Architects

• Business Fact Model provides foundation for Logical Data Model.
• Definition of business terms provides attribute-level definitions and metadata.
• Data Mapping documentation ensures correct use of Physical Data Model.

Quality Assurance

• Data Mapping and Business Rule documentation provides ready-made test criteria.
• Data Quality Evaluation documentation provides expected results.

Developers

• Data Mapping documentation provides direction for development of procedures to move data into correct data structures.
• Business Rules provide basis for transformations, aggregations, and derivations needed to support reporting or other applications.

Project Managers

• Business Rule and Data Mapping documentation provides a foundation for future estimates and resource allocations.

IT Security and Data Governance

• The Business Rules Approach provides a vehicle for documenting other types of rules such as access levels, authorizations, and regulatory requirements at an atomic level.

Data Projects Requirements Analysis

By now, Alicia was more than convinced that the data project she would be working on would be very different from the application development projects she had worked on in the past.  She had a good understanding of the Business Rules Approach and its benefits for the business and the rest of the project team.  But then came the big question — How do I approach getting requirements for this project?  In the application development world, it starts with use cases and process modeling.  Where do we start in the data world?

Most projects in the enterprise data warehouse can fall into one of two main categories.  You will get to your requirements by working your way down from high-level general concepts into your data requirements, or you will start with specific requirements and work your way up to data requirements.  Which way to go depends upon the project at the time and factors such as Business Intelligence maturity (does the business already have a reporting system?) and source system concepts (is this a new source system or a legacy replacement?).

Top Down Approach — Source System Changes Mean Data Changes

Sometimes a source system changes, either because of a legacy system replacement, or because of software upgrades.  Either way, when your source system changes, the requirements for the data warehouse are mandated by the source system program and by any existing reporting requirements that must be maintained.  A lot of times you are told to "keep the lights on" and make sure nothing breaks as a result of source system changes.  You start at the top and create your requirements by documenting the difference between the old and new, performing gap analysis at each level of detail.  

For example:  A legacy system is replaced with a new version, and all current interfaces must be replaced.  There are many downstream users of the legacy data in the warehouse, and each must continue to get their analytics uninterrupted.  Your requirements are going to come from the top, and you will work your way down to keep everyone's reporting going.

Projects of this type may not allow for enhancements to current work.  In that case your project will focus on analyzing the impact of data changes on existing documented work and implemented systems.

Bottom Up Approach — New Business Initiatives

When your business changes, your requirements are mandated by the specific information needs coming out of that business change.  Most of the time your business will have information needs in the form of specific reports or interfaces to be provided, where you start at the end point (data delivery) and work your way back up to the data warehouse and then further back up to the data sources.  This very often means you are introducing new data sources and that will also have an impact on existing processes that must be analyzed.

For example:  An opportunity to open up a new line of business through all sales channels is enabled by modifying business and sales processes and adding to the product mix.  Sales finance must have reporting that supports this new line of business.  This means that you have to start with what the business is going to need to know about their sales channels, their new products, and their processes, and you analyze what changes you must make to deliver their reporting.

Projects of this type usually involve major enhancements while at the same time keeping existing reporting and analytics running as designed.  Additional information requirements often mean there will be significant data source changes.  The project will focus on analysis of the impact of data changes on existing documented work and implemented systems as well as integrating new data sources and implementing new reporting or analytical processes into the EDW environment.

Regardless of the approach, you will consistently produce the same business-oriented deliverables every time.

Alicia summed it up nicely:  So we have a repeatable, business-focused, detailed process for documenting requirements on a data project.  We design our requirements gathering approach depending upon the project type.  But what are the project deliverables we keep talking about?  Doesn't any system lifecycle methodology tell us what we should produce?

Data Project Product Deliverables

The standard methodology does require deliverables of all types in all phases.  But as we've said before, there is a difference between the application project and the data project, and the data project requires more.   

In fact — it's not just a project deliverable that's being produced.  The data project deliverables provide both project and product documentation.  Each task in the data project leads to a long-term piece of product documentation that must be maintained and re-used as changes to the EDW are required.  Let's see this in more detail…

Data Project Task

Product Deliverable …

Define Business Terms Not just a glossary Develop Business Fact Model Conceptual view of facts about the business, not an E-R diagram Analyze Business Concepts Analyze Gaps in Data Model Add Missing Concepts to Data Model Analyze Current Source Data Analyze Gaps Enhance Current Source Data for Gaps

Glossary Atomic level definitions Fact Model Logical Data Model Physical Data Model Source Metadata Target Metadata Data Mappings Source to Target Mapping Map Reports to MxD to Data Model Map Reports to Reporting Layer to Data Model Business Rule Documentation Interface Specification Inbound to Data Warehouse Outbound to end users or systems Business Intelligence Specifications Measures by Dimensions Chart (MxD) Other…

Glossary

A main task in the data project is the definition of every business term — we call it a glossary, but it is much deeper and richer than a simple glossary.  Knowing what your business terms mean and having them defined at the atomic level leads you to the development of a fact model, which leads you into the Logical Data Model, and finally to the Physical Data Model at implementation.

The detailed and robust glossary provides the foundation for the next layer of documentation, a Business Fact Model.

Fact Model

The development of the business fact model is essential in gaining the business approval of the data model.  The business fact model is non-technical and gives an unbiased business view of their world.  This deliverable builds confidence for the business stakeholders knowing that you understand their business, and they can see it reflected in a simple, real-world way.

A Business Fact Model is a simple high-level graphical representation of Facts.  A Fact is defined as two terms (nouns) related by a verb phrase.  Here is a simple Business Fact for a Hotel Reservation:

Once you have all the facts represented on your Business Fact Model, you will have a good representation of scope for your data project.  The Business Fact Model provides a good foundation for data requirements.

Data Models

The three main types of data models are Conceptual, Logical, and Physical.  Data models are often the responsibility of a Data Architect or Data Modeler, although in some organizations a DBA or Data Analyst will create data models.  A full discussion of data model deliverables is out of the scope of this paper, but their creation is essential to the success of a data project and should never be skipped.

A Conceptual Data Model (CDM) is solution-neutral and business-oriented, much like the Business Fact Model.  We have been successful at using Business Fact Models in place of a CDM and also have created a CDM as an artifact in addition to the Business Fact Model.  Regardless of what you choose to do, you should always have a business-oriented, solution-neutral model such as a CDM or a Business Fact Model for your business stakeholders to review and approve.

A Logical Data Model (LDM) is still solution-neutral and will show the attributes, full relationships, and cardinality for the business concepts.  It can be a good tool to use with Business Stakeholders, depending upon the project.

A Physical Data Model (PDM) is created with all the physical details needed to build an actual database in a specific DBMS platform.  It is rarely reviewed and approved by business stakeholders.

Metadata

Metadata is associated with the data model steps and often is created by and stored within data model tools or metadata repositories.

At the very least, every data project should maintain basic logical and physical metadata, including attribute names, business definitions, and physical attributes such as data type, size, and precision.

Data Mappings

Once a data model has been created then mappings from source to target must be documented and maintained.  Data mappings can be used by the business and developers alike and can be done at all levels of detail, from logical to physical and at each interface point.  Interface points are defined as each transformation step that data must take from any source to any target.

A good data mapping will show:

• What data from the source is contained in the EDW;
• What data from the source is not contained in the EDW;
• What disparate data sources have been integrated into a single generic EDW structure;
• What data from the EDW is aggregated to a report or external system.

Data mapping documents and repositories are also the ideal place to document any business rules used and implemented for transformations, calculations, and aggregations in and out of the EDW.  Rules themselves can be named and mapped just like any data, and this gives future projects the ability to reuse the rules by referencing existing documentation.

This key piece of documentation should be done using a holistic approach, and should not be fragmented into spreadsheet tools if at all possible.  Data Mappings and Metadata can be combined into a single repository and can be governed by the business.

Interface Specifications

Interface specifications provide the technical information needed to create software interfaces used to move data between systems.  These can be done in a multitude of formats.  It is helpful if your interface specifications are standardized.

Business rules that have been documented in data mappings should be referenced in Interface Specifications but should not be duplicated into them.

Business Intelligence (BI) Specifications

Reports must be specified as interfaces are specified.  Again, this can be done in a multitude of formats, but if you can standardize on this it will be helpful.

• Measures by Dimensions Chart (MxD)

One key piece of information that we have found to be helpful in the analysis of reports and determining data gaps is to work with the business to create a Measures by Dimensions chart.  This simple chart will list every business concept that can be measured, such as:  revenue, unit volume, counts.  The chart then cross-references each measure against dimensions such as time and geography to create a matrix.  This simple little matrix gives the report designers everything they need for creating the correct data set and level of detail.

Specific business rules used by BI should also be referenced, but not duplicated.  BI tools usually have the ability to create metadata, and this can be leveraged to add to the metadata and data mapping documentation for the project.

Methodology and Standards are Key

We can say for sure that having a Business Rules Approach to a Data Project helps us complete our project successfully and completely.  No matter what you do, your methodology and standards are a key part of that success.  The advantage of providing standard documents is:

• Increase developer productivity:  Your developers know what to expect each time from each project.  If the EDW is an evolving, growing, and changing environment, then you will find having standards a real advantage.  Standardized documentation can provide a template for development tools and can automate code generation.
  
  
• Increase business stakeholder confidence:  The chances are your business stakeholders are the same people from project to project.  Once they know what to expect your future project acceptance process will become easier.  And for any new business stakeholders, having standardized documentation allows you to build an easy presentation and training template to make the process more efficient.

Data mapping is the single most valuable deliverable in a data project.  Without it your developers do not know where things go in the warehouse and out to the users.  Documenting the rules along with the data mappings provides a robust set of instructions about transforming, aggregating, deriving, or simply moving data out to the business.  So while it's important to have a standard, it's also important to have guiding principles that can be relied upon — to show commitment.  These things take quite a lot of resources, and if IT shows the business that these things are valuable then it makes absolutely no sense to shortcut or ignore them.  A set of guiding principles helps ensure that the process becomes automatic and is accounted for in all estimates and resource allocations.

As Alicia and I worked our way through this journey of sharing and learning, we were left with some observations:

• Data Projects Are Different … Not More Complex

There's no more complexity to a Data Project than any other type of project, but if you don't think about all the differences then you will certainly have difficulty.

• The Business Does Not Know Data … They Just Think They Do

Sounds kind of harsh … but our experience has shown that just providing data sometimes means that business users who thought they were reporting their information correctly were often not.  When the right analysis was done and documentation provided to the business showed the gaps, we were able to increase confidence in the reporting even though a period of adjustment needed to occur.

• The Business Will Resist the Business Rules Approach … Until They Understand It

Using a Business Rules Approach takes time and discipline.  This becomes resource intensive, so we needed to carefully explain the approach and its advantages to business stakeholders at the beginning in order to sell them on the process.  The extra effort pays dividends and helps ensure success.

• Follow Your Methodology … Enhance It For The Data Project

Don't think that the Business Rules Approach is a substitute for your methodology.  But you need to make sure you pay attention to the differences for a Data Project and enhance your methodology with the Data Project deliverables so that you don't miss any vital requirements.

• Be Consistent With Deliverables … Your Development Team Will Thank You

They thanked us by being more productive, and by improving their own level of code documentation.  It was a clear bonus to the overall result.

# # #