Six major components of test automation framework (Softwaregenius.com)

A test automation infrastructure, or framework, consists of test tools, equipment, test scripts, procedures, and people needed to make test automation efficient and effective. The creation and maintenance of a test automation framework are key to the success of any test automation project within an organization.

The implementation of an automation framework generally requires an automation test group. The responsibility of this group is to develop test automation infrastructure, test libraries, and tests tools.


The idea behind an automation infrastructure is to ensure the following:a) Different test tools and equipment are coordinated to work together.
b) The library of the existing test case scripts can be reused for different test projects, thus minimizing the duplication of development effort.
c) Nobody creates test scripts in their own ways.
d) Consistency is maintained across test scripts.
e) The test suite automation process is coordinated such that it is available just in time for regression testing.
f) People understand their responsibilities in automated testing.


Components of a typical test automation framework are as described in the following figure.


1) System to Be Tested: This is the first component of an automation infrastructure. The subsystems of the system to be tested must be stable; otherwise test automation will not be cost effective. All the subsystems must be stable and work together as a whole before the start of an automation test project.


2) Testing Platform: The testing platform and facilities, that is, the network setup on which the system will be tested, must be in place to carry out the test automation project. For example, a procedure to download the image of the SUT, configuration management utilities, servers, clients, routers, switches, and hubs are necessary to set up the automation environment to execute the test scripts.


3) Test Case Library: It is useful to compile libraries of reusable test steps of basic utilities to be used as the building blocks of automated test scripts. Each utility typically performs a distinct task to assist the automation of test cases. Examples of such utilities are ssh (secure shell) from client to server, exit from client to server, response capture, information extraction, rules for verdicts, verdict logging, error logging, cleanup, and setup.


4) Automated Testing Practices: The procedures describing how to automate test cases using test tools and test case libraries must be documented. A template of an automated test case is useful in order to have consistency across all the automated test cases developed by different engineers. A list of all the utilities and guidelines for using them will enable us to have better efficiency in test automation. In addition, the maintenance procedure for the library must be documented.


5) Testing Tools: Different types of tools are required for the development of test scripts. Examples of such tools are test automation tool, traffic generation tool, traffic monitoring tool, and support tool. The support tools include test factory, requirement analysis, defect tracking, and configuration management tools. Integration of test automation and support tools, such as defect tracking, is crucial for the automatic reporting of defects for failed test cases. Similarly, the test factory tool can generate automated test execution trends and result patterns.


6) Test Administrator: The automation framework administrator does the following

a) Manages test case libraries, test platforms, and test tools;
b) Maintains the inventory of templates;
c) Provides tutorials; and
d) Helps test engineers in writing test scripts using the test case libraries.
e) Provides tutorial assistance to the users of test tools and maintains a liaison with the tool vendors and the users.


Source: http://www.softwaretestinggenius.com/articalDetails.php?qry=957

Configuration Management Audit...

There are two meanings for the project management process of configuration management.

1. It can be used for the process of identifying, tracking, and managing of all the physical assets of a project. The items that you track under configuration management are called “configuration items” in the Capability Maturity Model (CMMI). 
2. It can also refer to the process of identifying, tracking and managing of all the characteristics of the assets of a project. These characteristics can also be referred to as product “metadata.” This is closer to the definition of configuration management in the Project Management Body of Knowledge (PMBOK®) from the Project Management Institute.

The following model describes the five major aspects of configuration management.

Planning. You need to plan ahead to create the processes, procedures, tools, files, and databases for managing the project assets or the metadata. You also may need to gain an agreement on exactly what assets are important, how you will define them, how they will be categorized, classified, numbered, reported, etc. The results of this up-front planning are documented in a Configuration Management Plan.

Part of your planning process should be to assign configuration tracking numbers to each type of configuration item.

Tracking. It’s important to understand the baseline for all configuration items. In other words, for each configuration item, you need to understand what you have at the beginning of the project. In many cases, you may have nothing to start with. In other cases, like physical assets, you may have some assets to begin with. The purpose of your tracking processes is to ensure that you can track all changes to a configuration item throughout the project.

You need processes and systems designed to identify when assets are assigned to your project, where they go, what becomes of them, who is responsible for them and how they’re disposed of. Since a project has a beginning and end, ultimately all the assets need to go somewhere. This could be in a final deliverable, into the operations/support area, scrapped, etc. You should be able to dissect each major deliverable of the project and show where all the pieces and parts came from, and where they reside after the project ends.

Managing. Managing assets means ensuring that they’re secure, protected, and used for the right purposes. For example, it doesn’t do any good to track purchased assets that your project doesn’t need in the first place. Also, your tracking system may show expensive components sitting in an unsecured storage room, but is that really the proper place for them? Managing assets has to do with acquiring what you need and only what you need. You also have to make sure you have the right assets at the right place at the right time.

Reporting. You need to be able to report on the project assets, usually in terms of what you have and where they are, as well as financial reporting that can show cost, budget, depreciation, etc.

Auditing. Auditing involves validating that the actual configuration elements (whatever they are) at any given time are the same as what you expect. Many projects get in trouble when they start to lose track of physical assets (for instance, material, supplies, code or other configuration items) or if the physical characteristics (metadata) of your deliverables is different that what you expect.

The auditing process is used to validate that the configuration elements match up with your expectations. These expectations are based on the original baseline, plus any change requests that you have processed up to the current time.

Source: http://blogs.techrepublic.com.com/tech-manager/?p=370&tag=content;leftCol

Stat tools - when to use what

1. Box Plots: used to tell if the data set is skewed by looking at the relative positions of the median, quartiles and the tail. Easy to spot outliers.
2. Scatter Plots: when we are interested in the relationships between two attributes. Offer visual assessment.
3. Control Charts: to know whether the data is within acceptable bounds.
4. Correlation: measure the correlation between two variables. Measure the strength of relationship / association.
5. Linear Regression: express relation as a linear formula.
6. Multivariate Regression: investigate relationship between one dependent variable and two or more independent variables.

M&A Needs

M and A requires you to identify needs:

1. Management needs
2. Technical needs
3. Project needs
4. Process improvement needs
5. Product needs

To start an M and A program, identify what your management requires as measures from the following categories:

a. Schedule & progress
b. Resources & cost
c. Product size & stability
d. Product quality
e. Process Performance
f. Technology Effectiveness
g. Customer Satisfaction

1. Once you have established these measures, you might need to be more granular, may lead to further decomposition of the measure
2. You will have to review the measurements as some of it might not be as useful you might have thought earlier

Sources of Variation

Example: How many text messages on an average a 19-year-old sends in a day.

1. Natural variation: Not all students will send the same number of text messages in a day. Taking one sample will not give all the information we need.
2. Explainable variation: Are there any other reasons why asking one student will not give the average for all 19 years? There are different types of students. Girls seem to send more texts and maybe rich students send more tickets. This is explainable variation.
3. Sampling error: I take a sample of 5 students. You take a sample of 5 students. Do you think the same mean/average will be the same? We will likely not, because we will choose different people. There are infinite different samples we can take from a population. Sampling error exists because a sample is not a true representative of the entire population. Sample size matters; the larger the sample size greater is the confidence.
4. Biased sampling: Bias happens when you pick up your samples based on prejudices like not including poor students in the above example.