Regression Testing...

Regression Testing Guidelines

1. REGRESSION TESTING – PROCESS 3
1.1 PERFORM IMPACT ANALYSIS 3
1.2 ESTABLISH TEST OBJECTIVES 3
1.3 CREATE REGRESSION TEST PLAN 3
1.4 IDENTIFY/CREATE REGRESSION TEST CASES 3
1.5 ENSURE REGRESSION TESTING ENVIRONMENT 4
1.6 EXECUTE REGRESSION TESTING 4
1.7 REVIEW REGRESSION TEST RESULTS 4
1.8 PREPARE TEST REPORT 4
1.9 EXIT CRITERIA 4
1.10 MEASUREMENTS 5
2. TOOLS FOR REGRESSION TESTING 5
3. METRICS FOR REGRESSION TESTING 5


1. Regression Testing – Process
A. Perform Impact Analysis
B. Establish Test Objectives
C. Create Regression Test Plan
D. Identify/Create Regression Test Cases
E. Ensure Regression Testing Environment
F. Execute Regression Testing
G. Review Regression Test Results
H. Prepare Test Report
I. Exit Criteria
J. Measurements

1.1 Perform Impact Analysis

The Team shall understand new requirements and analyze the impact of the added functionality to the existing features. Based on the analysis the impacted functionalities and new functionalities shall be identified for testing.

1.2 Establish Test Objectives

The Team shall Study and review each new requirement and Identify “key" System functions that need to be tested. Test objectives for each function shall be specified, Automation scope and tool shall be identified.

1.3 Create Regression Test Plan

The TL shall prepare the Regression Test Plan based on the new functionality as a result of the enhancements and the impacted areas. Scope shall be specified (the features shall be tested), Test Criteria gives the Entry and Exit criteria shall be defined. The resources requirements shall be identified. Responsibilities to the resources shall be assigned. Metrics to be captured shall be identified.

1.4 Identify/Create Regression Test Cases

The Test Designer shall identify a representative sample / subset of test cases to test impacted functionalities and prepare test cases for new functional scenarios.

1.5 Ensure Regression Testing Environment

The TL shall ensure the required test environment for Regression testing as identified in RTP. It is recommended that this setup is separate from the development environment. The smoke test shall be executed to ascertain if the build is stable and it can be considered for further testing.

1.6 Execute Regression Testing

Regression Test Cases shall be executed by the Tester manually and/or automated based on Project Requirements. All Regression Test runs shall be recorded in the Regression Test Cases/Test Report. If the observed behavior matches with the expected behavior, then “P” (Pass) shall be entered in the observed behavior column; otherwise “F” (Fail) shall be entered.

For every run, observed behavior shall be recorded. Test runs shall be repeated until all the defects are closed. All defects shall be recorded by the concerned tester. Defect Reporting and Closing shall be done using Defect tracking Tool.

1.7 Review Regression Test Results

The Regression Test results shall be reviewed by the TL. The TL may assign the task of review within the team. Reviewer may take some sample test cases and conduct testing to verify the correctness. The reviewer shall verify the following:

• All the relevant test cases are executed
• The choice of test cases have covered the main scenarios
• The testing is as per the corresponding test plan
• Identified defects are captured and analyzed

1.8 Prepare Test Report

The TL shall prepare the Test Summary Report after evaluation. Metrics specified in the test plan shall be prepared and analyzed.
Test Summary Report shall cover Test Execution and Defect Data summary. Based on the completion criteria, testing shall be signed off by the Test Manager.

1.9 Exit Criteria

The regression testing is completed when all the criteria mentioned in the agreement is met.

The basis on which regression testing can considered to be finished are
• When all the testing cycles are covered
• When the enhancements/changes are found to be met

A proper Signoff document should be prepared.

1.10 Measurements

Use the following metrics of Regression Testing to know effectiveness of this procedure:

• Defect Slippage
• Defect removal Efficiency
• Effort Estimation Variance
• Schedule Estimation Variance
• Productivity
• Defect Detection Rate

2. Tools for Regression Testing

The following tools are recommended for automated Regression.
• Mercury WinRunner
• Rational Robot
• Mercury Quick Test Professional
• Silk Test

3. Metrics for Regression Testing

Capture the following metrics for Regression Testing.
• Phase wise Defect removal efficiency and Total defect removal efficiency
• Failure rate/ Test accuracy/ Defect density
• Completeness/ Test coverage/ Defect indices
• Run reliability/ Error distribution/ Fault-days number
• Requirements traceability/ Cause and effect graphing
• No. of conflicting requirements/ No. of entries and exits/ module
• Review Effectiveness/ Delivered Defect Density
• Testing Effectiveness/ Testing Efficiency Test Case Design Effectiveness
• Phase wise Defect Density and Cumulative defect density
• Cost of Poor Quality/ Cost of Quality

CobiT

Control Objectives for Information and Related Technology (CobiT) is an IT governance control framework that helps organisations meet business challenges in the areas of regulatory compliance, risk management and aligning IT strategy with organisational goals.

Structure of CobiT

CobiT recognises 34 IT processes that are grouped into four domains. The four domains are:

1. Plan and Organise (PO)
2. Acquire and Implement (AI)
3. Deliver and Support (DS)
4. Monitor and Evaluate (ME)

Each process has a level of maturity (numerical) from 0-5. (0 is non-existent and 5 is optimised.) This scale can be used for a number of key evaluations, such as the level of maturity a process is currently at within your organisation, what level of maturity the processes should be at, what level is considered best practice, & what level the best of your competitors/other organisations have achieved.

BUSINESS AND IT CONTROLS

The enterprise’s system of internal controls impacts IT at three levels:

• At the executive management level, business objectives are set, policies are established and decisions are made on how to deploy and manage the resources of the enterprise to execute the enterprise strategy. The overall approach to governance and control is established by the board and communicated throughout the enterprise. The IT control environment is directed by this top-level set of objectives and policies.
• At the business process level, controls are applied to specific business activities. Most business processes are automated and integrated with IT application systems, resulting in many of the controls at this level being automated as well. These controls are known as application controls. However, some controls within the business process remain as manual procedures, such as
authorisation for transactions, separation of duties and manual reconciliations. Therefore, controls at the business process level are a combination of manual controls operated by the business and automated business and application controls. Both are the responsibility of the business to define and manage, although the application controls require the IT function to support their design and development.
• To support the business processes, IT provides IT services, usually in a shared service to many business processes, as many of the development and operational IT processes are provided to the whole enterprise, and much of the IT infrastructure is provided as a common service (e.g., networks, databases, operating systems and storage). The controls applied to all IT service activities are known as IT general controls. The reliable operation of these general controls is necessary for reliance to be placed on application controls. For example, poor change management could jeopardise (accidentally or deliberately) the reliability of automated
integrity checks.

IT GENERAL CONTROLS AND APPLICATION CONTROLS

General controls are controls embedded in IT processes and services. Examples include:

• Systems development
• Change management
• Security
• Computer operations
Controls embedded in business process applications are commonly referred to as application controls. Examples include:
• Completeness
• Accuracy
• Validity
• Authorisation
• Segregation of duties

Ref: http://www.itgovernance.co.uk/cobit.aspx

Project Estimations during proposal phase

Estimations during proposal phase.

Impact analysis is done, the application scope is determined to understand what all applications are within and out of scope depending on which upstream and downstream applications are identified. These have significant bearing on integration and end to end testing.

1. Correct estimations are pre requisite to successful project management
2. Accurate project estimates for a proposal ensure that -

• Project estimates are not too high, a proposal is not rejected.
• Effort and schedule estimates are not too low, thereby avoiding cost overruns and schedule delays in the proejct

When is estimation done:

• On receipt of a RFP from client
• Immediately on project initiation
• Beginning of each phase of the project (this varies for the type of project - Development, Maintenance, Conversion)
• Upon receiving CRs or Requests for Enhancements

Estimation @ RFP stage:

• Based on data given in the RFP
• Less data / knowledge available
• Low level details are not available
• May cover the following:-Estimate for each business function of the total requirements-Add overheads for quality assurance, project management-Estimate efforts for new technology, new tools, etc-Contingency

What to Estimate?

Estimate the Size, which then leads to estimation of schedule and effort.

In what units is Size estimated?

Size of work product is estimated in:

• KLOC
• FP
• Feature Points
• Number of Requirements
• Number of Components
• Estimation based on Use Cases for Object Oriented Projects

How is Effort derived from Size?

Productivity figures are taken into consideration. Also, COCOMO is used. Overheads for other activities are added (for example, User documentation, project management, quality assurance, user training, etc.). Note that in all these cases, complexity of the system to be designed and technology used must also be taken into account.

How is Schedule derived from Size?

T(in months) = y * 2 sqrt Effort in Person Years

• y is typically 2.5 to 3.0 for projects of effort <=180 person months
• y is typically 2. to 2.5 for projects of effort > 180 person months

While deriving Schedule take into account the leaves, trainings, and contingency.

What other things have to be estimated for?

• Team Size, Critical Computer Resources, Quality Assurance
• What factors affect estimations?
• Programmer experience
• Project documentation
• Standards
• Training Effort
• Contingencies
• Assumptions and Dependencies
• Management time
• Attrition, etc.

Other things to be factored in

• Knowledge and experience of the estimator (skill level of the estimator)
• Level of details available for estimates
• Time spent on understanding requirements and estimating
• Method used for estimation

Finally, what steps are involved in Estimation?

1. Identify the complexity level of each work item, obtain the figures for effort ratio for various phases and get the productivity figures for various phases
2. Use an appropriate estimation model, finalize the estimate for each phase of the project.
3. Document methods and calculations used for estimation, assumptions made, constraints taken into account, and loading plan
4. Consider experience of the team in the application area, exp of the team with the programming language, use of tools and s/w engg practices, and required schedule
5. Monitor these estimates continuously to track any change in the assumptions, constraints and loading plan

Elicit and Analyse Requirements

Understand and communicate requirements that align yout IT priorities with your business needs.

No process is more fundamental than the process of defining and managing business and technical requirements. It's no surprise that studies cite inaccurate, incomplete, and mismanaged requirements as the primary reason for project failure.

The Requirements-engineering process consists of two major domains: definition and management.

Best practices:

Elicit Requiremements:

Define the vision and project scope.
Identify the appropriate stakeholders.
Select champions (Voice of the customer).
Choose elicitation techniques (workshops, questionnaires, surveys, individual interviews).
Explore user scenarios.

Analyse Requirements: verify they are complete and achievable

Create analysis models.
Build and evaluate prototypes.
Prioritize requirements.

Specify Requirements:

Look for ambiguities.
Store requirements in a database.
Trace requirements into design, code, and tests.

Validate Requirements:

Review the requirements through a formal peer review.
Create test cases from requirements.

Manage Requirements:

Manage versions.
Adopt a change control process.
Perform requirements change impact analysis.
Store requirements attributes.
Track the status of each requirement.
Applying requirements best practices lead to higher satisfaction for your customers.

Courtesy:http://software-quality.blogspot.com/Achieve Useful Requirements: Matt Klassens @ Processes.http://www.ftponline.com/special/alm/mklassen/default.aspx

Implementing Configuration Management for Software Testing Projects

To analyze test process performance, testers typically review and analyze the test process artifacts produced and used during a project cycle. However, these testing artifacts, along with their related use cases, evolve during a project cycle and can frequently have multiple versions by project end. Hence, analysis of the process performance from different perspectives requires that testers know exactly which versions of artifacts they used for different tasks. For example, to analyze why the test effort estimates were not sufficiently accurate, testers need the initial versions of use cases, test analysis, and test design specifications they used as a basis for the effort estimation. In contrast, a causal analysis of software defects missed in testing requires testers to have the latest versions of use cases, test analysis, and test design specifications used in test execution.

Read rest of the article here: http://www.stsc.hill.af.mil/crosstalk/2005/07/0507Boycan.html
Courtesy: STSC (Software Technology Support Center)