The role of Testing in the Extreme Programming (XP) (Agile) Environment

by sergey on 14/12/06 at 6:54 pm

QA Consultants team typically utilizes a structured methodology that maps directly to the RUP approach while taking full advantage of the innovative Test Driven Development method. Iterative QA testing begins early and frequently during the development lifecycle of the application. Software quality assurance tests are carried out along three quality dimensions: reliability, functionality and performance.

Testing Phases

With a constant focus on quality QA Consultants testing efforts are fully integrated into software quality aussrance solutions and can be divided into the following phases:

Test-Driven Development

In QA Consultants’ solution Testing and Quality Assurance is an integral part of their approach. Traditionally occurring only at the end of the software development lifecycle, testing is also integrated into requirements analysis and development phases phase of QA Consultants innovative solution. QA Consultants’ testing strategy involves a team process that includes not only the quality assurance group but also the analysis and development groups. QA Consultants utilizes a comprehensive, multi-layer approach to internal system testing prior to the release the product to the Functional and UAT Testing groups for review and validation.

Crucial to this multi-layer approach is the use of test-driven development. QA Consultants does not defer testing until the end of the project. Instead QA Consultants shows how to begin it prior to writing the first line of code and make it an integral part of the development process. This means that before developers code a new piece of functionality they define and develop the unit test cases describing the required functionality. They continue to develop the code module until all unit test cases succeed.

Here are the Quality Assurance benefits of the Test Driven Development that QA Consultants draws upon during the project:

· Integrate testing into development and requirement analysis in order to prevent “too little too late” testing

· Improve design by reducing coupling and improving cohesion

· Easily support refactoring

· Avoid unclear requirements and focus developers on writing code to meet specific requirements

In order to capitalize on the above benefits QA Consultants recomends an investment into the Test-Driven Development method right from the beginning of the Development stage. QA Consultants proven processes are strict and well defined; in this innovative approach through test automation and continuous integration, this investment is kept cost effective. QA Consultants experience clearly demonstrates that QA Consultants find majority of the Severe and Critical issues during the Development stage, thus allowing the Testing and UAT stages to be much smoother and significantly faster.

Coded Functional Tests (CFTs)

These tests are performed by analysts to test the interaction of the user interface with the middle-tier of the application. CFTs are based on the acceptance test cases and simulate business scenarios. While CFTs do not test user interface behavior, they cover a substantial portion of an application and are more efficient and less fragile than the UI level test scripts that follow. Successful completion of CFTs

Wherever possible, as indicated by the name, CFTs are coded and automated. This allows the testing automation and continuous integration processes leverage CFTs in the same manner as unit tests.

Functional End-to-End Testing

There are seven major phases of activities associated with QA Consultants testing methodology and the resulting testing workflow:

Test Planning

This activity consists of two tasks, planning and design.

Testing efforts need to address approach, strategy, staffing, equipment, environment, scheduling, etc. which need to be coordinated.

Planning for testing automation should also begin early in the development cycle. Proper planning for test automation is essential for building a maintainable, reusable, reliable, and scaleable process. Automation and automated tools can be used for functional testing, performance testing, and load and stress testing.

Test Case/Test Script Development

Test cases are detailed instructions for test engineers that explain how to test one or more test conditions. These test cases are directly derived from use case scenarios, main flows and alternate flows. To increase efficiency and avoid duplication of effort Testing/QA organization will leverage Coded Functional Tests that were created and executed by business analysts during the Construction phase.

Test Environment Preparation Phase

The goal of this activity is to determine the testing infrastructure required to support the test effort. A test environment is the collection of physical files, programs, databases, and screen-ware required for a test to execute properly. The preparation includes environment preparation activities, including operational and infrastructure support, as well as developing the installation, backup, restoration, and construction procedures. In addition to the development of the test environment(s), the test team should determine data states, data recovery points, and critical points for test restorations.

Test Execution

The main task associated with test execution consists of applying the test cases against the application solution for validation purposes. This can be done manually or in an automated fashion like using tools. The test engineer executes the test cases identified by the Test Plan, documents the results, and validates those results against performance criteria.

Test Results Analysis

The test results analysis activities of the testing lifecycle involves the evaluation of the test results and the comparison of these results against the acceptance and certification criteria previously established in the test plan. Some types of analysis that may be reported include:

· Flow Analysis – confirms that the flow of events occurred as expected. Do the tasks outlined within the technical requirements flow as anticipated?

· Failure Analysis – used to report how the application performed in regards to failures. Usually, some existing keywords in the results log can indicate an anomaly or failure. Look for those keywords and examine the events preceding and following the keyword to categorize the failure. Continue this analysis until the entire log has been searched or it is determined that the suspension criteria have been met. With the use of automation, pass and fail criteria can be automatically logged.

· Fulfillment Analysis – used to identify which test cases indicate fulfillment of the requirements. Look for keywords that indicate that an event occurred as desired. Examine the events around the keywords in the results log to insure that the application behaved as expected.

· Measurement and/ Metric Analysis – analyze the test result measurements and metrics that will be contained in the Executive Management Application Readiness Report.

Management Reporting

The final activity of the testing lifecycle is management reporting. The objective is to communicate the conclusions and recommendations of the test team to Project Management, the Project Team, and to the User Community as appropriate

The test methodology described above reflects a proven process for testing and implementing software applications. It is however only one portion of QA Consultants methodology with a proven track record.

Performance Testing and Engineering

A critical success factor of a software implementation project is its ability to meet performance expectations for execution flow, response times, and operational reliability and limits. Performance testing and engineering are used to ensure the establishment of a solution and a class of tests that are implemented and executed to characterize and evaluate the solution.

Performance Testing

QA Consultants advocate the early adoption of a suite of performance testing as early as possible in the project life cycle in an effort to minimize the impact of invalid architecture assumptions. The early adoption of theses tests is facilitated by the automation of CFTs, mentioned above, which require only a moderate effort to write a load testing harness to execute as a set of automated functional tests. QA Consultants Performance testing suite would typically include:

· Benchmark Testing – uses a standard reference workload to measure the performance of a new or unknown system and compare it to a known reference system or measurement.

· Performance Testing – uses a constant workload and varying system variables to tune or improve the performance of the system. Measurements typically include the number of transactions per minute, number of users and size of the database being accessed.

· Load Testing – verifies and assesses acceptability of the operational limits of a system under varying workloads while the system-under-test remains constant. Measurements include the characteristics of the workload and response time. When systems incorporate distributed architectures or load balancing, special tests are performed to confirm that the distribution and load balancing methods function appropriately.

· Stress Testing – confirms that the system functions as intended when abnormal conditions are encountered. Stresses on the system may include extreme workloads, insufficient memory, and unavailable services/hardware or diminished shared resources.

· Volume Testing – focuses on the ability of the system to handle large amounts of data, either as input, output or resident within the database. Volume testing includes test strategies such as creating queries that [would] return the entire contents of the database, or have so many restrictions that no data is returned, or data entry of the maximum amount of data in each field.

· Scalability Testing – determines that the site performance does not degrade as the number of concurrent users approaches the level, or above the level, specified in the load requirements and during every scenario that it can be anticipated to experience.

· Stability Testing – System stability is verified by operating the system for a long period of time under normal operating load and under tightening security without observing any degradation of performance.

QA Consultants team focuses on two critical areas to achieve high levels of application and system performance. These are:

· Designing to prevent “bottlenecks”

· The ongoing evaluation and capacity prediction of the application’s supporting infrastructure

Services that address the first area fall under Performance Engineering, and are normally applied during all phases of a development lifecycle. Services that address the second are Performance Management, and are directed at analyzing ongoing server, database, network components, network facilities and application capacity performance.

Performance Engineering

Performance Engineering has four objectives:

1. The simulation and modeling of response times, throughput and resource consumption while influencing application design and implementation decisions.

2. The analysis and identification of real and potential performance issues arising from architecture and implementation decisions.

3. Identification and Instrumentation of key performance metrics and statistics for ongoing tracking and predictive analysis.

4. The initial sizing of workstations, servers, database components, network components and facilities.

In contrast to merely tuning an application so that it meets performance requirements, QA Consultants team recommends that simulation and modeling techniques be employed throughout the development process so that product selection and design decisions can be made early in the lifecycle.

The application is profiled at multiple levels including:

User Interface

· Application rules

- Specific method invocations

- Execution of application rules

- Calls to third-party components or external modules

· Database access

- Selects, inserts, update, deletes, stored procedures, DDL commands

· External Interfaces

- Invoking legacy applications

Adjustments to product selection and design late in an application’s lifecycle to correct performance issues are costly and will cause schedule delays. QA Consultants recommends using third-party tools such as JProbe for profiling the application. These Profiler tools help developers quickly diagnose performance bottlenecks in the Java code. With accurate diagnostics, including line-by-line results, developers can dramatically improve application and system performance.

The use of measurements and metrics should be used as a guide to answer pertinent quality questions. The Goal – Question – Metric paradigm is useful in developing measurements that are based on goals and answer questions about the goals. The metric is the low level item that is expected to help provide the answers. Typically, a combination of metrics produces a measurement and the associated answers to the goal question.

The most common categories of metrics are as follows:

· Test Coverage – these measurements answer questions about how many requirements have been tested. The answer to those questions does not indicate how many tests have passed or failed. This can be represented as a snapshot percentage or displayed as a trend.

· Error Discovery – these measurements answer questions about when defects are detected and how many defects have been detected. The defects may be sorted by severity or represented as a total.

· Open Defects – these measurements answer questions about when defects are open and how many defects are open. The defects may be sorted by severity or represented as a total.

· Testing Activity – these measurements answer the questions about how much testing has occurred. The answer to that question does not indicate how many tests have passed or failed. This can be represented as a snapshot percentage or displayed as a trend.

· Pass Status – these measurements answer the questions about how many tests have passed and how much of the Web Services Solution is working correctly.

· Specific Risks – these measurements answer the questions about which tests have failed and how important they are in regards to the release of the solution.

Regression Testing

Typically, due to a large number of internal and external (3^rd party) components that projects interact with, QA Consultants strategy is to employ the regression test suites written for these applications. QA Consultants experience has proven that capitalizing on the existent investments in this area is not only cheaper and more efficient but is also much more productive as existent regression test suites are usually written by people with deep understanding and knowledge of these systems and applications.

Program Interdependencies

In order to facilitate a significant number of concurrent and interdependent programs that often go into production before or during the release of a project, QA Consultants relies on the PMO team to facilitate a smooth and efficient coordination of these activities while QA Consultants utilize the CFT regression test package that is created during initial stages of the QA Consultants Test-First Development process (please see above).

UAT

QA Consultants innovative methodology utilizing allows for business and end-users to be interactively involved in the Elaboration and Construction phases of the Development Life Cycle. Such process as Interactive Reviews, close relationship between developers and business and Iterative Development Approach that allows for team reviews and approvals every two weeks. All of these factors allow for Business and End-Users to provide feed often thus allowing for a “no-surprises” UAT cycle that in turn becomes much shorter and then a common UAT phase.

The QA Consultants view of testing is composed of three key areas: methodology, tools, and resources.

Triangle Testing Success Figure

The testing methodology described above forms one of the dimensions of the success triangle. The second dimension is QA expert Consultation knowledge of major test tool vendors such as IBM-Rational and Mercury Interactive. This knowledge allows QA Consultants to stay abreast of technology capabilities through the latest product releases and philosophies. The third dimension represents the quality management resources that QA Consultants brings to the engagement.