Software Quality Assurance Blog

Here is a link to an artical about Canad’s leading software QA Testing company at the Backbone Magazine

Author and industry analyst Ian Hayes and Compuware’s Mike Koza will provide an overview of the CIT concept and quantify how CIT can benefit both development and testing organizations. Ian Hayes also is writing a white paper along the same topic, which will be made available immediately following the QA webcast.

CIT for .NET
Tuesday, January 23
11 a.m. PST/2 p.m. EST
Register now »

CIT for Java
Thursday, January 25
11 a.m. PST/2 p.m. EST
Register now »

What is needed as a minimum?
Each data warehouse load job should be developed and tested incrementally throughout its development life cycle:

• Testing that the correct data is being extracted from the source
• Testing the transformation logic
  Performing source to target validation
• Testing the error-handling logic
• Testing the data throughput rates
• The incremental testing is followed by integration testing – in which the separate jobs are scheduled together into an interdependent sequence and re-executed
• The analytics applications themselves must be tested against the data-warehouse to ensure delivery of the correct data
• In order for data warehouse QA project to succeed, the following has to be in place:
  A test team
• An independent test environment
• Support from Business Analysts, ETL developers, DBA and management
  While assessing the QA practices at a client as far they relate to data warehouse testing, these are the items which QA Consultants looks at.

QA Team Lead

• Is the DW QA team lead skilled enough to:
  Prepare project plan for testing efforts
  Evaluate staffing needs
• Provide size estimate for QA environment
  Has solid knowledge of data warehouse concepts and testing intricacies associated with business intelligence applications
  Strong enough and yet diplomatic to oppose pressure from development, business analyst and project management where quality sacrifices are required
• Strong enough technically to be able to address complex issues and to review work being done by the testers
• Able to review defects on the regular basis and perform root cause analysis
  Where and when necessary come up with processes and procedures and enforce them on the team
• Testing skills of DW QA team
  Do the team members understand basics of testing methodologies such as levels of testing – unit, system, regression, etc
  they purely black box testers, or can they also do white and grey box testing
  they have leadership strong enough to plan, size and distribute the work
• Is the test team leadership strong enough to oppose the pressures from development, business analysts and project management to “sweep things under the carpet”
• Is the test team in control of QA environment and code promotion
  the support of developers, business analysts, system administrator and DBA secured during the off-hours testing
• Can team members provide sizing for the test environment – data base and the file system

QA Environment

• Are the test carried in isolated test environment
• How closely does the environment replicate the actual production environment
• Does the QA team have control over the test environment, i.e. the code to be tested is pulled into QA environment rather then pushed by developers
• Is the test data properly backed up
• Are the snapshots taken at appropriate times to allow roll-back to a particular state
• Is the environment large enough to accommodate the data size for testing
• Can test environment easily be adopted for performance/stress/load testing

Project Environment

There may be issues at the project level that affect performance of the QA team, but be outside of QA team control

• Is QA involved in the process at an early stage
• Are business requirements, functional specifications and data model clearly defined and signed off by users, business analysts, developers and testers
• Is a change management process clearly defined, communicated and strictly adhered to especially with respect to signed off requirements
• Is a code freeze implemented when required and all subsequent code changes are either a result of defect fixes or approved change requests
• Is the promotion to production done from QA environment
  Potential Problems

QA Consultants –findings may point to existence of some of the following problems:

• Users are not sufficiently engaged
• Testing reports - not data
• Garbage in, garbage out
• Skipping the reconciliation of data warehouse data with source system data
• Relying only on  “made up” test data
• Underestimating the testing workload
• Failing to set proper expectations with business users
• Fast tracking” the testing phase.
• Underestimating the duration of the ETL process.
• Choosing the “wrong” testers.
• Skipping proper sign-offs

QA Consultants © 1994 – 2007 - A Plaza Consulting Inc. Company. 

Objective

• Unit test is the test of individual units. 
• The objective of the unit test is to ensure that each individual component works correctly in isolation according to any applicable requirements, as defined in the design documents.
• The following processes must be satisfied before Unit Testing can be executed efficiently.
• Developer has to check that following criteria are satisfied:
• Outstanding programming issues have been resolved
• All program code compiles cleanly and has been reviewed and documented appropriately
  Standards and procedures for unit test have been communicated to the team
  Individual development test environments must be set up

Conditions

It is the expectation that the developer will perform the following type of tests:

• Positive/Negative Tests
• Bug Fixes
• Responsible Team
• Development Team

QA Consultants © 1994 – 2007 - A Plaza Consulting Inc. Company. 

Objective
The objective of performance test is to validate the back-end architecture, hardware and applications scalability:

• Determine the performance, stability and scalability of an application under various load conditions.
• Determine which configuration sizing provides the best performance level.
  Determine if the current architecture can support the application at peak user levels.
• Prove application is stable enough to go into production (Acceptance).
  Determine if the new version of the software adversely had an impact on response time.
• Determine at what point does degradation performance occur (Capacity Planning).
• Identify application and infrastructure bottlenecks.
• Evaluate product and/or hardware to determine if it can handle projected load volumes.

Conditions

• Passed assembly testing
• Responsible Team
• Infrastructure Team

Objective

System test aims to test the system as a whole, only those aspects related to the whole system will be tested. The main objective of the system test is to verify that the users’ business requirements are met and to ensure the validity of the following aspects in the application’s design:

• Overall system functionality, end-to-end business functionality works as expected

User interface

• Integration with external systems
• Security
• Technical architecture components
• Report failures to development team so defects can be identified and fixed

Conditions

• Passed assembly testing\
• Responsible Team
• Technical Team
• Test (Q/A)

QA Consultants recognizes the critical need for solid performance testing processes for development and deployment of mission critical applications.  Application availability / reliability, user experience, and business performance depend on applications operating effectively within a predicted service activity model.  QA Consultants has successfully implemented a performance test program on numerous client engagements and has developed a core competency in performance engineering.  While our typical implementation approach leverages the flexibility and power of Mercury’s LoadRunner product we do have resources skilled in all of the industry leading performance test platforms:  Empirix, IBM Rational, Segue, etc.

QA Consultants leverages industry best practices by conducting performance testing in three phases:

1. Performance Profiling
2. Load Testing
3. Stress Testing

Performance profiling is a performance test in which response times, transaction rates, and other time sensitive requirements are measured and evaluated.  The goal of Performance Profiling is to verify performance requirements have been achieved. Performance profiling is implemented and executed to profile and tune a target-of-test’s performance behaviors as a function of conditions such as workload or hardware configurations.

Load testing is a performance test which subjects the target-of-test to varying workloads to measure and evaluate the performance behaviors and ability of the target-of-test to continue to function properly under these different workloads.   The goal of load testing is to determine and ensure that the system functions properly beyond the expected maximum workload.  Additionally, load testing evaluates the performance characteristics (response times, transaction rates, and other time sensitive issues).

Stress testing is a type of performance test implemented and executed to find errors due to low resources or competition for resources. Low memory or disk space may reveal defects in the target-of-test that aren’t apparent under normal conditions. Other defects might results from competition for shared resource like database locks or network bandwidth. Stress testing can also be used to identify the peak workload the target-of-test can handle.

Benefits of Investing in Quality

• Deploy business-to-business and business-to-consumer applications with confidence
• Maintain tight project deadlines by avoiding rework
• Accurately model performance behavior, plan for scalability and identify potential bottlenecks
• Maintain company prestige and reputation among consumers by delivering a quality online customer experience.

The QA Consultants Difference

Our combination of technical expertise and business knowledge uniquely positions us to help our corporate clients successfully use new technologies to meet business goals.

• Our level of assistance is tuned to what you need and how you work.  We can do a turn-key test, or teach your people how to use new processes, methods and tools.  
• Our approach stresses “rules before tools” . We believe that the disciplines learned over the past 10 years provide the foundation for success.
• Our approach emphasizes an integrated team – schedule pressures  won’t allow for a separate QA team that doesn’t share the deadlines and goals of the project team
• QA Consultants has formed alliances with the leading performance test tool vendors, allowing us fast access to highly skilled technicians and special pricing on software licenses for our clients.

Various terms are used to describe the different aspects of performance testing.  Teams that are creating new applications, or modifying existing applications, should be aware of the following:

• End-to-end Test  - Before performance testing can take place, it’s important to exercise all component and interface functionality starting with the browser and extending all the way to back-office and legacy functions.
• Load Test - Some hardware configurations use load-balancing software to distribute incoming traffic across multiple web-servers.  Load testing uses special tools and techniques to validate the correct functioning of these components when subject to high volume conditions. 
• Performance Test - This term refers to the validation of external and internal application behavior under varying input conditions.  Performance tests will be selected that vary multiple criteria, such as number of concurrent connections, type and volume of transactions,  and/or volume of data throughput.   The goal of performance testing is to stress the system to the breaking point and find weak links. 
• Reliability Test - These types of tests are run for longer periods (e.g. 2 to 5 days) than typical performance tests, and are designed to detect problems such as memory leaks which can degrade performance over longer periods of time.
• Scalability Test - A scalability test is an advanced performance test that attempts to predict how the application will perform under various “what-if” conditions.  Scalability tests are typically used to model system behavior in order to gain confidence before peak usage conditions (e.g. High traffic holiday shopping).
• Stress Test - An alternative name for Performance testing. 
• Volume Test - A performance test that emphasizes high volume of data throughput.  This type of test is designed to exercise database and data transfer components.
  

© QA Consultants

It is common for developers to rush eBusiness applications to production and forgo performance and scalability testing.  Things may run fine at first, but an increase in traffic may cause unexpected problems. 

Some real-world problems that we’ve seen at our clients include:

• Web/application server performance problems due to underestimating processor requirements.
• Performance problems due to errors implementing load balancing.
• Scalability of web and application servers, and uptime problems associated with their stability.
• Performance bottlenecks due to errors implementing middlware.
• Functionality errors in end-to-end transactions due to problems with interfaces to ERP and legacy systems.

Problems such as these can occur at any layer in the n-tier environments that today’s applications are being developed for.