SQA Methodology and Processes

Document Version: V1.1

Date: 05/16/2000

 

 

 

 

 

SIGN-OFF SHEET

The following approvals are required to initiate the COMPANY Testing Methodology and Process document for all future quality assurance projects:

 

Approval Name:

Department

Approval Signature:

Date Approved:
 

VP of Engineering

    

Frank Lara

Engineering – Quality Assurance

    
 

Engineering - Development

    
 

Engineering – Development

    
 

VP Operations

    
 

Technical Support

    
 

Engineering - CTO

    

 

Other Project Participants for Distribution Only:

Development Team

Quality Assurance Team

Project Managers

Program Managers

Product Managers

 

 

SIGN-OFF SHEET *

Preface *

Purpose *

Audience *

Document History *

1.0 Project Description *

2.0 Deliverable Items *

2.1 Documents *

3.0 Development, Test and Production Environments *

3.0.1 Code Migration across different environments *

4.0 Test Data *

4.1. Engineering Environments *

4.2 Production Environments *

5.0 COMPANY Product Development Process *

5.1 Request for Enhancements or Functional Specifications *

5.2 Walkthroughs and Requirements Analysis *

5.3 Design Phase *

5.4 Coding or Implementation Phase *

5.5 Unit Testing Phase *

5.6 Configuration Management and Control Phase *

5.7 Preliminary QA *

5.8 Formal QA Testing Phase *

5.10 Internal Beta Testing Phase *

5.11 User Acceptance Testing Phase *

5.12 Release To Production *

6.0 Test Methodology *

6.1 General *

6.2 Test Configuration *

6.3 Requirements Analysis and Ambiguities *

6.4 Build Verification *

6.4 Build Verification *

6.5 Preliminary QA - RFEs and New Functionality *

6.6 Defect Regression Testing *

6.7 Functional Regression Testing *

6.8 Performance Testing *

6.9 Test Standards *

7.0 Test Automation *

8.0 Test Management and Control *

8.1 Project Problems *

8.2 Test Logs *

8.3 Configuration Management *

9.0 Acceptance Criteria *

10.0 Emergency Fixes *

10.1 Data Problems *

10.2 Software Problems *

11.0 Production Problems Maintenance *

12. 0 Special Feature Releases/Emergencies *

APPENDIX A – COMPANY Engineering Testing Strategies *

APPENDIX B - QA Test Standards – Test Plan/Test Cases *

APPENDIX C – FAST TRACK FLOW CHART *

APPENDIX D – Test Case Log – Sample *

APPENDIX E – QA Documentation Numbering System *

Appendix F – Code Migration and Test Environments *

Appendix G – Detailed Test Environments *

 

Preface

This is the Quality Assurance Testing Methodology and Process document.

Purpose

The purpose of this document is to describe the COMPANY - Engineering group internal quality assurance methodology and testing process.

Audience

This plan is intended for use by the Engineering project teams to manage development and QA testing processes and to ensure that a quality product is developed before installing in a production environment.

Document History

Version

Date

Who

Revision

V1.0

9/2/99

FAL

  

V1.2

5/00

FAL

Modifications to process

 

1.0 Project Description

This Quality Assurance Plan describes the process for the development, test and verification of the COMPANY products. The objective of this plan is to provide a planned documented approach towards implementing specific development and testing methods which will be utilized in the development of COMPANY products.

2.0 Deliverable Items

The following documents will be written and placed under Configuration Management and Document Control. Each document will also have a related document template to standardize the way project documentation will be written:

2.1 Documents

· Marketing Requirements Document

· Functional Specifications

· Configuration Management Plan

· Development Design Plans for Major Revisions

· Test Plans for Major Revisions

· Test Matrix

· Test Cases

· Test Scenarios (Workflow)

· Test Logs for the Release

· Defect Reports documenting each Release

· Release Notes for each Release

· User Documentation

  1. Development, Test and Production Environments

 

3.0.1 Code Migration across different environments

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3.0.2 QA Test Environments (Excluding International Projects)

 

  1. Test Data

4.1. Engineering Environments

The Engineering group will manage test data generated by the Development and QA Teams. Periodically, test data will have to be refreshed to maintain a quality development and test environment.

4.2 Production Environments

Test data generated in the IT staging server and production environments (Demo, First Look and Production) will be managed by the Information Technology group. Periodically, test data will have to be refreshed to maintain a quality production environment. The IT group will manage test data, the use of projects and the control of access.

 

 

5.0 COMPANY Product Development Process

    1. Request for Enhancements or Functional Specifications

Based on either RFEs from the Product Marketing or defects identified by team members or external individuals, an item is entered into the tracking system with a status. Product Marketing defines a release and Engineering, consisting of RFEs and old defects with a priority assigned. At the point that the release is defined, the team leader of the developers will assign tasks for the release to individual developers.

For RFEs, the Product Marketing Manager writes the Marketing Requirements Document. The requirements define the business needs of the customers. The Program Manager will then take these requirements and develop a functional specification document that describes the changes required for the feature. Also at this time, a Project Plan/Schedule is written by Marketing, with Engineering input, to identify and schedule all the tasks for the release.

 

5.2 Walkthroughs and Requirements Analysis

RFEs and/or functional specifications are then reviewed by the Engineering Team members assigned to this item (Developers and QA). Initially, the Program Manager will host a walkthrough of the documentation to describe the changes or enhancements. This walkthrough will ensure that the impacted members of the Project Team understand the proposed changes or enhancements. A requirements analysis is conducted by QA, with an assigned Developer, to determine if any ambiguities or issues are uncovered in the documentation. Ambiguities or issues found are then returned to the Program Manager for resolution. A number of iterations may be required before the RFE(s) and/or functional specification are approved for implementation and identified for a specific release. The final approved project documentation is then reviewed by the Project Team member(s) and frozen. SofTest is the tool used by the QA Team to conduct an Ambiguity Review (Requirements Analysis).

 

5.3 Design Phase

In the design phase the application development team will review the project documentation for the release and develop a design document for projects that are complex or contain interdependencies or external interfaces. Once a design document has been created, the developer will host a design walkthrough that is attended by other developers, QA, and optionally other interested parties. The purpose of these reviews is to discuss the technical aspects of the RFE(s) and/or functional specifications and determine the design approach for the development team.

Ideally, this phase is the last opportunity to introduce any changes or new functionality into the release and the last chance to uncover the majority of ambiguities in the RFE or functional specification. This is not to say that additional minor ambiguities will not be discovered during the coding or testing phase, but at a minimum all major issues should have been identified by this time.

In addition, during this phase, the QA Team is busy finalizing the Test Plan, Test Cases and Work Flows for this release. Test Plans are written for major projects only. At this time, walkthroughs are also conducted (major projects) of the Test Plan, Test Cases and Work Flows. If walkthroughs are not conducted, then each QA Team member is expected to get other members within QA Team to review Test Cases or other test documentation.

 

    1. Coding or Implementation Phase

The coding or implementation phase is the actual development of the code for the proposed changes or enhancements. During this phase, developers will conduct walkthroughs of the proposed code changes or enhancements. For smaller projects, these walkthroughs may be somewhat informal and primarily consist of desk checking by one of more peer developers. For larger projects, a more formal code walkthrough will be conducted. While most RFEs will have been introduced prior to this phase, it is still possible that critical business needs, or refinement or clarification of requirements may require changes during this phase. At the end of this phase, there is code completion. Code completion means that all design expectations were meant and coded.

 

5.5 Unit Testing Phase

Prior to checking the code into Configuration Control, each developer is responsible for testing the code. The developer builds the code and integrates it with the most current baseline stream created by Configuration Control. At this point the developer tests the integrated application code to ensure that the fixes to known system problems, feature enhancements or new functionality were completed according to the requirements and did not adversely impact other areas of the application. If unit testing identifies any errors or deficiencies, additional coding is done.

At this time, the QA Team may conduct an informal or preliminary QA of the RFE, if the code is available early on, during the coding phase and before formal QA testing begins. Also, if the change is identified as having a heavy impact on current application functionality, a preliminary regression test of the entire product may be conducted.

 

5.6 Configuration Management and Control Phase

After the defect, RFE or new functionality is coded, the developer checks the code into Configuration Control. The Configuration Control person will build a test build of all code checked in by all developers. At the completion of building the test build the Configuration Control person will change the status of all code checked into Ready For QA.

The Quality Assurance Team will then test the test build to determine if the build is suitable for further testing. If the test build passes, then the Configuration Control person will build a formal test baseline for further testing by the QA Team.

 

    1. Preliminary QA

The informal or preliminary QA is the first opportunity for QA, Project Managers, Product Managers or anyone involved with the project to do a test drive of the product. This test can be done on the development server (internal), QA server (internal) or on the Sneak Peek server (external) in Production. There is no guarantee that the feature under scrutiny will not be riddled with defects. This approach will give customers a chance to determine, are we building what was wanted. It is also gives QA an opportunity to start testing and flush defects out, early on.

 

5.8 Formal QA Testing Phase

The QA Testing Phase begins when the Configuration Control person has completed building the formal test baseline that contains all of the changes and enhancements to the application or feature. The code for the release or feature is code complete. During this phase the QA Team is testing RFEs, new functionality and any defects defined for the release. In addition and after initial testing is completed, the QA Team conducts a regression test, to determine if any modifications to the application code breaks existing application functionality.

This phase is an iterative process. As testing proceeds RFEs, new functionality or defects will be changed from Ready For QA to QA Verified ( Verified – Production Ready for items going straight into Production) or QA Rejected, in the defect tracking system. If the modifications fail, then the entire development process is repeated.

This testing methodology is further described in section 6.0 - Test Methodology. This section presents a detailed explanation of the testing process.

At the completion of the formal application-testing phase, the following deliverables are available for review, by Project Team members:

· Test Plan

· Test Matrix

· Test Cases

· Test Scenarios (Work Flow)

· QA Test Logs/Results

· Defect Summary

· CM Release Notes

· Percent Complete Test Cases

 

5.10 Internal Beta Testing Phase

Concurrent with the QA Testing phase, the internal Beta Team executes a subset of the QA test cases written and any user oriented tests. The internal Beta Team is responsible for determining whether the defined release is ready and suitable for release to the customer and user acceptance.

Beta Team members are primarily Project Managers and Product Managers, but other members may be selected to participate. At the completion of this effort, a sign-off is required from the Beta Team and the release and code is now frozen and handed off to IT. IT puts the release in a secured IT staging area and prepares the release for production installation.

Release Notes and related project documentation is gathered to package this release for Project Manager implementation and Technical Support.

    1. User Acceptance Testing Phase

The User Acceptance Testing Phase is the final customer acceptance of the feature designed, coded, tested and delivered. If the customer was involved from the beginning of this release, there will probably be only minor issues to be dealt with. This is the final sign-off of the release by the customer, before the release is moved into the Production server, from the Demo server.

Prior to UAT, a checkout of the release is done, on the Demo server, to ensure that the delivered release is functional.

5.12 Release To Production

The IT group will now move the release from Demo into the Production environment. A release checkout will be conducted again to ensure that the delivered release is functional.

6.0 Test Methodology

Refer to the following diagram and text for a detailed explanation of the QA Team’s test methodology.

 

 

6.1 General

Testing a new product feature is an iterative process. An entire suite of Test Cases or Test Scenarios (Workflow) will be executed and defects will be recorded and reported, as they are found. Defects will be fixed, another build will be created and another iteration of testing will be conducted. This testing effort will be executed again (regression tested), to ensure that a fix was confirmed and that the fix did not break some other part of the system. This iterative process will continue until the product is considered stable enough to run under a production environment.

Siebel will be used as the defect tracking system. All testers will use this system to record and report defects found. The control and migration of the software will be managed by the Configuration Control person (See Software Configuration Management Plan).

 
    1. Test Configuration

      2.  

      1. Client Hardware

        2.  

      3. Server Hardware
        4.

        See Automation Plan for server configurations.

         

         

      4. Client Browser Software

Product

Version

Manufacture

IE

4.72.2106.8

MS

IE

5.0

MS

NS

4.08

Netscape

NS

4.5

Netscape

 

6.3 Requirements Analysis and Ambiguities

The QA Team, with development, will conduct a requirements analysis of the functional/requirements specification or RFE. After the analysis, the QA Team, will provide a list of ambiguities. SoftTest and Visio will be used as the tool to do the ambiguity analysis.

The ambiguities list will be distributed to the Program Manager to provide answers or resolutions. This process will be iterative and will continue until all ambiguities are resolved, before the development of any application code.

6.4 Build Verification

A Basic Assurance Test will be done by a member of the QA Team for each test build created by the Configuration Control person. The purpose of the B.A.T. is to ensure that the test build is suitable enough for QA testing.

The Configuration Control person will also maintain release notes for each build created. These release notes will contain the current version of the build and related changes to the product. At the completion of the B.A.T. the QA Team Leader or designee will inform the Configuration Control person that the build passed B.A.T. and is ready to be built into a formal release for the QA Team to conduct further functional testing.

6.5 Preliminary QA - RFEs and New Functionality

The initial focus of testing the new release is to test all new RFEs and/or new functionality of the release. This approach will reveal any defects that may result because of modifications to the application. These defects are recorded into the defect tracking system and the developer is informed immediately and given the opportunity to resolve the problem.

During the preliminary or informal QA phase of testing, QA Team members are busy working directly with the developer to do an informal QA of the RFE. This usually occurs during developer unit testing. This will allow the developer to get an initial quick reaction to the changes being made in the application, before formal QA begins. The difference between formal and informal QA is that defects found during informal QA are not recorded.

6.6 Defect Regression Testing

At the completion of testing modifications to the application, the QA Team will then focus on testing defects that are assigned to the current release, then re-tested again during each iteration of regression testing. This will ensure that any defects that were re-coded do not break existing functionality and that previously fixed defects that were fix confirmed are still functional.

6.7 Functional Regression Testing

Regression testing of current application functionality is then tested by executing all of the baseline test cases. This type of testing will ensure that current application functionality was not corrupted by new modifications to the application. If problems are found during this phase of testing, defects are recorded in the defect tracking system and assigned to the development team for resolution.

A functional regression test may be partially executed during the early stages of development, for critical changes to the application that may impact current application functionality. A complete regression functional test is executed on the formal baseline that is identified as the final release to the IT. A complete regression functional test does not occur for each release, unless changes may effect the global functionality of the product under test.

6.8 Performance Testing

Performance requirements are not specifically delineated in the functional specification, but will be determined by the Engineering Team. Optimum performance of the integrated system is a goal of the QA Team, but not acceptance criteria. Performance testing and improvement may require the coordinated efforts of the Engineering Team related to servers, network, and products and application code as required. This is described more in the QA Automation Plan.

 
    1. Test Standards

The QA Team will develop testing standards. These standards will be used for the design of Test Cases, Test Scenarios, Test Procedures and Customer Workflow documentation.

Testing standards will be developed for the following:

· Test Plans

· Feature Set – Test Matrix or Check List

· Feature Set - Test Cases, Test Procedures, Customer Workflow

· COMPANY Regression Check List

· Test Logging

· Defect Reporting/Graphs (Per Release/Feature)

· Release Notes

· Test Summary: % Complete, Defect Summary

See QA directory Osiris – Quality Assurance – Standards.

7.0 Test Automation

Initially all testing will be done manually. Test automation will occur after the feature has become stable enough to minimize the changes required in maintaining automated Test Scripts. A group of Test Cases will be selected, as portions of the product become frozen. These Test Cases, along with some manual testing, will be used for the purpose of conducting regression testing. Silk will be the tool to be used for automating most of the testing effort. As changes occur to the functionality of the product, automated Test Scripts will also require changes.

The automation and maintenance of the application is a full time effort by one or two FTEs, therefore this effort will be dependent upon the availability of a full time resource dictated to automation and not testing the application. Eventually, the entire QA Team will train in automation of the product. See the Automation Plan for a detailed explanation of automation.

8.0 Test Management and Control

 

8.1 Project Problems

Any ambiguities, issues, RFEs and defects will be entered into the defect tracking system. These problem reports will then be reviewed and filtered by the application QA/DEV Team meetings. The QA Team Leader will provide all the necessary reports related to project issues, RFEs, defects. The QA Team Leader will facilitate QA/DEV Team meeting. Problem reports will be reviewed and categorized as ambiguities, issues, RFEs or defects. Defects will be then be assigned severity levels and priorities. All ambiguities, issues, RFEs and defects will be assigned to someone on the team for resolution. Team Leaders will be responsible for giving status on assigned problems. Refer to the Defect Control Tracking document for the process regarding defect control and management.

 

8.2 Test Logs

Test Logs will be kept during the process of testing for the release and stored for future reference. All test execution results and test logs our kept in the QA Team automated test case database. See Appendix D.

8.3 Configuration Management

Software will be checked in and out as changes occur, using adopted CM tools. Builds for each baseline will be created by the Configuration Control person. Builds that go into a production environment will be considered a formal release of the product. Developers will provide the Configuration Control person with Release Notes , for each build, indicating the specific changes to any modules. The defect tracking system will be updated based on this information. The Configuration Management Plan describes the process for managing this effort.

 

9.0 Acceptance Criteria

The acceptance criteria for the application are the successful execution of related product functions, as stated in the functional specifications or RFE and demonstrated by the successful execution of the created Test Cases or Test Scenarios (Workflow).

 

10.0 Emergency Fixes

All emergency problems found in Production take precedence over any current development in process. The emergency must be a showstopper and keeps the customer down and unable to proceed with project work. The point of contact for emergency problems, in Production, is the Technical Support Manager or designee. All emergency fixes will follow the normal QA process, fix and test on the QA server, and finally in Demo or Production.

    1. Data Problems

Data problems are fixed immediately by the assigned maintenance development engineer. The Technical Support Manager addressing these problems directly with the responsible development manager to get it assigned and fixed.

 

10.2 Software Problems

If these problems are minor fixes, similar to a data problem, then the Technical Support Manager addresses this problem directly with the responsible development manager to get it assigned and fixed.

If the Production problem is a major issue and needs to take priority over data problems and minor emergency fixes, then the Technical Support Manager addressing the problem with the Director of QA and the Director of QA pushes the fix through development, with the assistance of the development managers.

 

11.0 Production Problems Maintenance


Production maintenance issues list was created to develop a process to clean-up current documented problems, in Production.

Technical Support owns the development of this list. This list is given to the Director of QA and will be responsible for working with Technical Support and development to get them assigned and fixed, as soon as possible. The Director of QA will provide weekly status to Technical Support and the rest of the COMPANY Team, as appropriate.

The weekly status meeting will provide status on the all Production fixes. All fixes will follow the normal QA process, fix and test on the QA server, then finally in Demo or in Production.

 

12. 0 Special Feature Releases/Emergencies

This process has a more serious impact on Engineering activities. In order to get approval to implement this process, the VP of Engineering must approve the fix or project. In order to approve this some of the following needs to be considered:

· Current release schedule activities

· To work on something new, a feature or schedule will have to be

adjusted

· Customer impact or income considerations

· Use across the board for COMPANY customers

 

13.0 Rapid development projects

The rapid development testing process is a compromise of quality and speed, and is now the adopted testing process for projects on the fast track. Quality assurance has two main purposes. The first purpose is to assure that the product released has an acceptable level of quality. The second function of quality assurance is to detect errors at the stage when they are least time-consuming (and least costly) to correct. The longer an error remains in the product, the more time-consuming (and more costly) it will be to correct.

The success of the rapid development process consists of two elements:

  1. Choosing effective practices that lend themselves to the rapid development process
  2. Choosing practices that are oriented specifically toward achieving the scheduled objectives.

Rapid Development (or Fast Track Projects – FTP) is schedule oriented (i.e.: beating your competitors to market). One of the most straightforward ways to save time on a software project is to modify the process to avoid doing things twice.

Some basic components required for FTP to occur:

  1. The product life cycle (PLC) shouldn’t extend beyond 30 days or 4 weeks
  2. At least one member of each team and/or department involved in the project needs to be represented in the FTP team (also known as SWAT team)
  3. A minimum list of Marketing requirements or MRD
  4. Minimum technical specifications and screen shots of the UI (not necessarily frozen)
  5. Test Cases (TC) and Test Plans (TP) provided by QA and signed by PMs and Marketing
  6. The GUI is frozen half way through the product life cycle
  7. Code frozen is attained one week before delivery as to provide QA with enough time to do a final functionality test and get input from the rest of the company thus allowing developers to fix any serious bugs arising at the last moment. NOTE: P1 and P2 must be fixed before the application can be signed off for deployment to COMPANY.
  8. A minimum level of stress and load test is performed once the application is deployed in the production environment.

See Appendix C for the diagrammed flow of the testing fast track model.

 

APPENDIX A – COMPANY Engineering Testing Strategies

This is a list of general testing strategies and definitions to be used in the overall testing and quality assurance effort.

AUTOMATED BASIC ASSURANCE TEST (Future)

The Automated Basic Assurance Test (A.B.A.T) is a subset of the tests executed for Functional Testing and will be done for each test build created by the Configuration Control person. The purpose of this testing to ensure that the QA Team has a build that is sufficiently successful to conduct further functional testing. It only certifies that the build was successful and further testing is required.

FUNCTIONAL AND DEFECT REGRESSION TESTING

At the completion of the A.B.A.T the QA Team will conduct regression testing efforts. This phase of testing involves the testing of defects that were fixed by the development team. Functional testing, consisting of the application test case testbed, will then be done to ensure that what was fixed did not break some other part of the application.

Functional testing is testing the application under test according to the functions described in the product External Functional Requirements Specification, if a project has one. Functional testing only proves that the requirements, as specified in the External Functional Requirements Specification, are met..

Functional testing also includes exercising additional various combinations or permutations of the function under test. This is analyzing the function and determining what sort of other tests can be executed, to thoroughly test the total functionality of the product, other than meeting the functional requirements of the specification or change request. This type of testing goes beyond meeting the requirements of the function and includes negative and ad hoc testing of the function. It requires the use of your imagination to create test cases that exercises the product beyond its intended functionality, in a number of different permutations or combinations. The intent here is not to cover every possible combination or permutation, but only the most obvious combinations or permutations of the function.

 

NEGATIVE TESTING

A good source of negative testing is to conduct testing by forcing errors to occur based on an error-processing document. The following negative testing are further suggestions:

1. Invalid use of keystrokes, mouse, function keys and miscellaneous keyboard operations.

2. Repeated operations between softkeys, navigation buttons and tool bars.

3. Invalid and valid interaction with Windows, if a window based product (Ex. - Maximizing and minimizing screens, navigating from one application to another, execute product-exit to windows- return to product, etc.).

4. Invalid selections of menu selections and menu selection function keys.

5. Inappropriate use of the function or set of functions.

6. Invalid data inputs (Ex - Put alpha characters in a numeric field or combination or blanks, or invalid length).

7. Insert more information in a field than allowed.

8. Unplug the machine or network when in the middle of executing the application.

9. Press other keys or attempt to do another operation while executing the application.

10. Decrease the amount of memory or system resources available for the application. There is software available that will allow you to do this.

ERROR PROCESSING TESTS

Some products, especially in a client server environment, interact with other applications, systems or other platforms (Client to middle tier to back end or server) and during this process the product may traverse through many different paths. This can be an exhaustive error-processing test. It is recommended that in this case the Test Engineer should first concentrate on their assigned application error processing first and then coordinate with other projects to ensure that the error processing related to their respective areas is conducted. However, the Test Engineer should not just rely on other projects to test errors that may relate to the assigned application when it is communicating with other parts of the system. Also the Test Engineer should read the error messages returned and determine if the error makes sense to the audience it is intended for. For example, error messages are different for users versus system programmers.

 

LIMITED PERFORMANCE TESTING

Performance testing involves the testing of the product under stress, maximizing the load of the application or volume testing and timing the ability of product to handle capacity or load over a specific period of time. See the Automation Plan for detailed explanation of performance testing.

AD HOC TESTING

Ad Hoc testing involves informal and undocumented testing of the product. Test it any way you wish. Pretend to be an unsophisticated user. Push buttons, navigate functions and menus without purpose. Attempt to break the system. In some cases informal test cases should be documented, if it makes a good test or the steps are required to duplicate a found defect.

SYSTEM TESTING

System testing involves testing the application without focusing on the functional aspects of the application and how the application interacts with other applications or features in a fully integrated environment. It takes into consideration how the product works with other interfaces outside the functionality of the product and how the product responds to unusual exercises. The following are some of the areas covered under system testing and only selected items that are relative to the testing effort of the application will be conducted::

Volume Testing

Subject the product to heavy volumes of data. The purpose of volume testing is to show that the product can not handle the volume of data specified in its objectives. For instance, a data bases may only be handle a limited number of tables or items within a table, so a test could be created to go beyond this capacity.

 

Stress Testing

Stress testing involves subjecting the product to heavy loads or stresses. Unlike volume testing, a heavy stress test is a peak volume of data encountered over a short span of time. If a product can only handle 10 simultaneous users on the system, then exercise the product with 11, 15, 20 users.

 

Performance Testing

Products sometimes have specific performance or efficiency objectives, stating such properties as response times and throughput rates under certain workload and configuration conditions. If these requirements are defined, then it is important to exercise these performance requirements as stated and beyond. For example, it may be stated that the product will retrieve customer data within 1 second after the search key is pressed.

Interface Testing

There may be other applications or a product may be part of a larger system. In this case the product must be tested against these other interfaces its communicating with or through. For instance, in a client server environment there may be interdependencies between one product and another that communicates through some other interface via the Internet. Also in a client server environment the test team must take into consideration the interfaces between the client, middle tier and the back end. Each piece of the client server environment is integrated separately, tested separately and then tested again when all of the different pieces are integrated to make a complete operational system.

Usability Testing

This type of testing involves the usability of the system or is the product designed well enough for humans to use (Human Factors). This type of testing requires the Test Engineer to critically study the design and use of the system and determine if the system is user friendly. These type of problems are submitted as change requests and not defects. Be tactful when trekking through this area, because egos can be at stake. This type of testing also involves testing the product, as the agent in the field may actually use it. The QA Team primarily focuses on meeting the functional requirements of the release and does a limited amount of usability testing. This testing effort is best suited for the Beta testers and the User Acceptance Test Team, in determining the usability of the product for the customer.

 

 

 

Security Testing

Security testing is the process of attempting to devise test cases that subvert system security checks. Security testing is a very important aspect of testing, especially when it involves a bank. Sometimes it may be better to hire an outside reputable organization that has experience in this area.

Configuration Testing

Configuration testing is testing the product under a different number of hardware or software configurations. Hardware and software can sometimes have an unlimited number of configurable items and this would be too much to test, so it is advisable that the product be tested with, at least, the minimum, the maximum configuration and a few different configurations in between.

Compatibility Testing

There are many products that are dependent on a number of other software platforms. For example, the introduction of an upgraded operating system specified that it was compatible with applications previously executed under the older operating system version. This required testing a number of applications with the new operating system. It is important to know what are the compatibility requirements of a product and understanding all the interdependencies or pieces that work with the product being developed. The bottom line is does the product still work with the old requirements and will it work with the new requirements. This is what upward compatibility is all about.

Installability Testing

The successful installation and execution of the product across different software and hardware configurations is an important part of testing the product. This type of test can be combined with configuration and compatibility testing. Sometimes the product may have a set of installation diskettes or installation procedures that require testing. The installation process should be documented in the early stages of the development process and put into the Configuration Management Plan. This will make the install process easier for the implementation phase of the development process.

Recovery Testing

This type of testing takes into consideration what will happen if there is a crash or other problem with the product. Can the system or product recover. Are there back-up procedures in place and do they work. Some of the problems can be a hardware failure, memory parity errors, I/O device errors or data errors. For example, unplug the network line in the middle of an operation, introduce noise as data input to a communication line, back up the system and introduce problems while back up is being executed.

Documentation Testing

Documentation testing involves testing any documentation the user may require. Such as, User Manuals, Reference document, Installation Procedures, etc. The Test Engineer must actually follow documentation instructions to the letter to determine if any problems occur. Documentation testing also covers any on-line Help screens, menus, tutorials or user messages.

 

APPENDIX B - QA Test Standards – Test Plan/Test Cases

 

 

 

 

 

 

 

 

NOTE: See Director of Quality Assurance or QA Team member for the standard. Also review the test documentation numbering system (Appendix E) for the storage and retrieval of this documentation.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

APPENDIX C – FAST TRACK FLOW CHART

 

 

APPENDIX D – Test Case Log – Sample

Test Engineer(s)

Start/Finish Date

Test Round

% Complete

Comments

 

 

        

Test Plan Approved:

Test Cases Selected:

Test Data and SW Installed:

Successful Smoke Test:

Functional Test Done:

Stress Test Done:

Performance Test Done:

COMPANY Regression Test Done:

Defects Reviewed:

Sign-Off Completed:

Defects Found:

P1:

P2:

P3:

P4:

Total Open:

Total Closed:

Grand Total:

Test Completion Approved by Director of QA:

      

Test Case ID

Function

Pass/Fail

Defect Number

Comments

1.NC001X.00

        

2.

        

3.

        

4.

        

5.

        

6.

        

7.

        

8.

        

9.

        

10.

        

11.

        

12.

        

13.

        

14.

        

15.

        

16.

        

17.

        

18.

        

19.

        

20.

        

21.

        

22.

        

23.

        

24.

        

25.

        

26.

        

27.

        

28.

        

29.

        

30.

        

 

 

APPENDIX E – QA Documentation Numbering System

 

Purpose

The purpose of this numbering standard is to prepare for the growth for the number of test cases being created for the COMPANY eMarkteplace quality assurance program. Sometime, in the future, QA will be acquiring a data base test case management tool. QA will use this tool to manage the growing number of test cases QA will write.

In addition, numbering QA test cases will facilitate managing results of executing these test cases. A test log will be created and the numbering system will be used to manage % complete and pass/fail results. Eventually all of this will be automated with exact results documented and there will be no need for verbal approximate estimates. If you have a problem coming up with a number, see me, also make sure you do not duplicate the numbering system. Make adjustments where necessary.

Test Plan Numbering System
  1. FMTP001.00 – FM = Feature, TP = Test Plan, 001 = Test Plan number, .00 = versioning number when making updates to the Test Plan.

Test Case or Test Matrix Numbering System

A variation is using the same set of steps in a Test Case, but the set-up may be different. For example, running the same Test Case and steps for a GC, then an Arch or an Owner. Sometimes it does not make sense to create an entire new Test Case, just to number a new variation. For example, you could write into your Test Cases all the steps, then at the bottom of the test case you could put, GC, Arch, etc. Use your best judgment and keep it simple, but follow this standard. If you are going to deviate from the standard, then you need to see me.

  1. FMCPY001X.00 – FM = Feature, CPY = Function, 001 = Test Case number, X=variation of the same Test Case being executed, but just a different variation (lower case a-z), .00= versioning number when making updates to Test Cases. If you do not have a variation for the Test Case do not use X.
  2. FMCPY001M.00 – FM = Feature, CPY = Function, 001 = Test Case number, capital M = This identifies this as a Test Case Matrix (spreadsheet) of many functions or variations to be executed, and 00 = versioning number when making updates to the Test Case Matrix.
  3. FM001X.00 – The difference between this numbering system and number 1 above, is that there is no one function that identifies this Test Case, because many functions are being exercised or this may be a series of functions being executed. So there is no function ID.
  4. SIT001X.00 – SI = System Integration Test. There is no specific Feature or Function to be executed. This is a Test Case that may be exercised across various COMPANY products. An example is Bidding with Content. If it is within a product, but across many Features, then use BPMSIT001X.00.

 

Other Examples

File Manager Feature:

Copy Function: FMCPY001X.00

Uploading Function: FMUL001X.00

File Locking Function: FMFL001X.00

Non-Conformance with NO functions identified: NC001X.00

Submittals with No functions identified: SUB001X.00

RFI with No functions identified: RFI001X.00

Global Setting with NO functions identified: GS001X.00

Field Observations with No functions identified: FO001X.00

Supplemental Instructions: SI001X.00

Cost Apps – Owner Issues with No functions identified : CAOI001X.00

Bidding:

Bidding Introduction: BIDBI001X.00

Bidding Manger: BIDBM001X.00