Agile Business Intelligence (BI) changes our perception of BI projects by dramatically changing their economics and execution. Instead of regarding them as something that 'the organization might start next quarter if they can line up the resources', they can be viewed as something that 'I can start this afternoon'.
As emphasized before(Make it work than make it better.html), the UltraVista's main business driver "Make It Work, than Make It Better" fits perfectly into the agile paradigm. As we are used to by now, the word "agile" is used as a buzzword in many contexts and in different ways. We are using the word "agile" in its traditional definition: marked by ready ability to move with quick easy grace (http://www.merriam-webster.com/dictionary/agile). So, by Agile BI, we mean 'the ability to create BI solutions quickly and easily, in a nimble and well-coordinated way'.
Using an agile approach improves the success rate of Business Intelligence projects. It does this by changing the economics, the technical solution, and the execution of the projects.
Agile and Lean Principles
In recent years organizations have been increasingly using agile and lean software development methodologies and tools. This rise in popularity is spurring the adoption of agile philosophies in other domains.
Adapting the principles of the Agile Manifesto to work with BI leads to these:
- Satisfy the customer through early and continuous delivery of valuable data and features.
- Welcome changing requirements, even late in development.
- Deliver a working solution frequently and measure progress by this.
- Foster a closer working relationship between business people and developers throughout the project.
- Build projects around motivated and knowledgeable individuals.
- Decide late, deliver fast.
- The frequent delivery of a working solution will obviously solve some of the problems BI projects face including:
- Communication and vision gaps will be reduced in each iteration as end users see the working results.
- Development latency will be significantly reduced.
- Shortcomings of the top-down or bottom-up approach will be alleviated as rapid iterations allow a hybrid approach that combines or alternates them.
You can reduce development tasks and costs by using the "decide late" principle. By treating the first delivery of a BI solution as temporary until proven otherwise, you avoid extra work and cost. Some examples of savings are:
- Use manual flat-file extracts from source systems instead of fully-automated data flows.
- Extract a partial (but still useful) set of data. The data can be limited by a time range or can be restricted to a subset of a geographical, organizational or other dimension. Make sure the extracted data is fully useful to a subset of users, not partially useful to all of them.
- Transform the data into simple fact tables instead of star, snowflake or other complex data schemas.
- Install the solution on existing hardware, or cloudbased hardware.
- Use open source databases, middleware, and front-end software instead of proprietary software.
- Don't bother with automation, auditing, production controls, etc.
- Monitor the usage of the system for a month or two.
Only if the system is still being used frequently after this period of time should you automate the data transformations, increase the scale of the data, optimize the performance, provision hardware, switch software and/or implement production controls and automation. Some organizations invest in these 'institutionalization' levels in phases that can span a year. This is not a case of trying to ignore, or hide, the long term costs of successful BI projects. It is a way to invest in BI projects incrementally as their value becomes proven. The advantages of an agile approach can be applied to different aspects of a BI project:
- Agile BI can be used to develop a straight-forward BI solution in its entirety
- Agile BI can be used to develop the requirements for a large scale project
- Agile BI can be used to investigate data quality or data integration issues
An agile BI team is typically made up of 4-5 people, each typically having one of these roles: IT Developer, Project Manager, BI Consultant, End User, Business Analyst, and/or a Database Administrator. Any of these people is capable of starting a project on their own.
Many spreadmarts in existence today are complicated and intricate. Most have been constructed by end-users because an officially sanctioned BI solution is neither available nor planned. This shows that there is a population of technically-oriented end-users who are willing and able to create BI solutions.
Having these individuals on the team and giving them tools that enable them to experiment will help BI projects significantly. Ideally the team should be based in the same location, and if they can work in the same room most of the time, that's even better. Regardless of location, the team should be provided with tools to help them collaborate, such as forums, mailing lists, wikis, and a document/content management system.
If you need to acquire computing hardware before a BI project can begin, you can run into trouble. In some cases it delays the start of a project, in other cases it is a contributing factor in a project's cancellation.
To get a project going quickly, or enable a prototype to be conducted cheaply, you'll find it advantageous to use one of the following:
- User hardware: Using existing desktops, workstations, or laptops means no procurement delays or budget spending. A desktop environment is great for a business analyst or technically-oriented end-user to get started on a project.
- Cloud computing: Cloud computing quickly and cheaply make a BI solution available to a distributed group of people. This includes both public clouds like Amazon EC2 and private clouds like Eucalyptus.
In some cases user's hardware is locked down and only certain applications are available to them such as office productivity, email, web, and corporate applications. In these case cloud computing gives technically-oriented end-users a new option.
In most cases a BI solution will go into production on dedicated, on-premise hardware. But prototyping and development can be done on desktop machines and cloud environments. The ability to migrate easily from user hardware to cloud environments, and cloud environments to static deployments further increases the productivity of the team and the flexibility of the project. For these hardware options to be viable, the BI software must be suitable (in terms of licensing and hardware requirements) for all those environments.