1

This Reverse Engineering Test Plan provides an outline of the reverse engineering testing strategy, between BIDDERCOMPANYNAME and ClientCompanyName, for the 128 encryption project.

1.2 Deliverable Documents

BIDDERCOMPANYNAME Certification Packages for itemized features to be tested (see appendix A)

1.3 Revision History

Date	Author	Revision History
3/30/2000	Frank Lara	Draft V1.0

Review and Sign-off

Department	Person	Date	Signature
Engineering
Quality Assurance
BIDDERCOMPANYNAME - Management

Scope

A table of features to be tested is attached to this document, Appendix A – Test Completion Schedule. This will define the scope of the reverse engineering and testing effort. These were provided by ClientCompanyName along with an initial top prioritization of ten specific features, noted in the "Notes" column of that table. The initial walkthrough of each feature provided the basis for sizing of individual testing efforts. A closer follow-up led to the estimate of hours required for reverse engineering, test case development, data setup and initial run of tests for each feature. The table points out several assumptions concerning the total project effort:

The total effort from analysis through the first testing should take inside of 800 hours

The top ten priority items should require about 325 hours, or 2 ½ weeks for the team

Many features are of limited functionality, and so each can be completed quickly

Certain groups of features can be analyzed and tested together, with a savings of time

Testing of the ClientCompanyName Admin features, as well as features such as the New Project Wizard, will require the set up of a complete project, for use only by the BIDDERCOMPANYNAME project team

An additional 150-200 hours can be anticipated for complete regression testing and final certification. This projects a total time frame of some eight weeks for the entire effort. A variable that is presently unknown in this projection is the total amount of time necessary for data setup. Unforeseen problems in dealing with sheer volume of testing, or problems related to ClientCompanyName specific data setup might extend completion of the project beyond this rough estimate. Every effort will be made to enlist the expertise of ClientCompanyName personnel in overcoming such problems as early and as easily as possible.

Reverse Engineering testing will be performed to verify that each function works as documented in the reverse engineered test scripts. BIDDERCOMPANYNAME will test functions at ClientCompanyName, as documented in the standard BIDDERCOMPANYNAME Certification Packages. BIDDERCOMPANYNAME and ClientCompanyName may select a sub-set of these test cases, to demonstrate that each feature of the 128 encryption build executes successfully, as specified by the reverse engineered test scripts.

A QA Summary and Defect Control List will be reviewed at this time, and the results of the initial reverse engineering testing effort will be reviewed.

A ClientCompanyName QA Engineer may work with the BIDDERCOMPANYNAME test team, during the execution of test cases, to become familiar with BIDDERCOMPANYNAME test cases and systme functionality.

At the end of this testing effort, a formal meeting will be conducted by the ClientCompanyName Program Manager and BIDDERCOMPANYNAME representative(s) to get a preliminary sign-off.

A formal sign-off will be done, at the successful completion of the ClientCompanyName regression test.

Test Data

CLIENTCOMPANYNAME is providing a set of "pre-packaged projects containing data from which tests can be run. If additional data is required, it will be generated as part of individual test setups. Testing timeframes assume that data can be reset as necessary and that new test data can be preserved. If data setup requirements prove difficult to the point of jeopardizing completing tests in a timely manner, the test manager will be notified.

Test Platform

The following are the hardware, operating systems and browsers that will be provided to the BIDDERCOMPANYNAME team. A sub-set of BPM test cases will be selected to be executed on the various platforms.

The BPM application will be loaded and configured on a ClientCompanyName QA test environment. An initial smoke test of the application will be conducted by BIDDERCOMPANYNAME to determine that the application is ready for further testing and formal user acceptance testing. The selected QA environment will contain the desired release of the BPM application. No changes will be made to that release without the approval of the test team in order to preserve the integrity of the testing.

5.1 OS Platforms

Operating Systems

Hardware Model

Manufacture

Modem

Windows 95

P5-350

Gateway

Dial-up-56K

Windows 98

Celeron 466

HP

Intranet

NT

P5-300

Gateway

Intranet

5.2 Client Browser Software

Product

Version

Manufacture

Description

Internet Explorer

HP Celeron 466-98

5.0

MS

Internet

Browser

Netscape

HP Celeron 466-98

4.7

Netscape

Internet

Browser

Functions To Be Tested

The list of functions to be tested is specified in the "Test Completion Schedule" (Appendix A) at the end of this document. Functional testing is expected to be conducted to two stages. For each of these features, testing will be restricted to functionality associated with the flow of work across the screens of the application. This would include rules for choosing one flow over another and navigation decisions.

Feature Project Schedule

Quality Assurance Tasks Target Start Date Target End Date

Develop Test Cases 03/21/00 04/21/00

Functional Testing

Initial execution of test cases 04/24/00 04/28/00

2^nd and 3^rd run of test cases 05/01/00 05/12/00 or 5/15/00-6/2/00

Manage Software Defects 03/21/00 05/12/00

ClientCompanyName Sign-off 05/12/00 05/12/00

Functions Not To Be Tested

The following functionality is not currently scheduled to be tested as part of this plan:

Field edits and validations associated with screen operations
Correctness of database updates and other process functions
Integrated functionality across major ClientCompanyName application components

The only functionality being tested under this plan is the exercising of workflow options and the associated screen navigation

Testing Approach

CLIENTCOMPANYNAME provided BIDDERCOMPANYNAME with a comprehensive "tour" of the application and documentation of most key functional areas. There are approximately ten Feature Sets each with multiple screens and associated processing functionality. Attached is a spreadsheet of the scope of functionality encompassed by this engagement. BIDDERCOMPANYNAME would develop test cases for each of these feature sets and screens using the following process:

Examine available documentation and execute some "trial cases" to confirm understanding of the functional behavior of the feature set.
Conduct brief, focused interviews with knowledgeable functional experts to result specific issues of intended system behavior. All such interviews will be fully documented so that the information is captured for future CLIENTCOMPANYNAME staff use.
Design a suite of functional test cases using BIDDERCOMPANYNAME’s automated test design tools and related procedures.
Review the test cases with CLIENTCOMPANYNAME for correctness.
Execute the test cases, record bugs and re-execute as required to confirm changes to applications.

Test cases will be organized into certification packages for each functional test suite. Test cases will be designed using RR/RT test modeling techniques and the automated design of detailed test scripts to cover all functional variations. Ambiguities in the expected behavior of the system will be identified during the design process and resolved as quickly as possible. If the expected behavior does not coincide with the behavior of the current version of the product, the differences will be either specified as an intended difference or logged as a defect.

An early focus will be to execute all test cases at least once to ascertain the general correctness of the release and to produce the first round of defects for correction. Additional test runs will be conducted as necessary until all test cases executed successfully. For planning purposes, the schedule will anticipate three such runs, including the first run. Overlap of test runs is expected as corrections will be made to each subsystem and released to QA in stages.

The functional testing needs to occur over the combinations of OS and Browsers listed in section 5.2. However it is not required that all tests be executed against all combinations of OS and Browsers. The approach will be to design and execute test cases using specific OS/Browser combinations for specific functional areas. Thus for any specific OS/Browser, full functional testing will be accomplished for specific features. Selected tests for the remaining features will be executed against the same OS/Browser configuration.

Testing will therefore be conducted in three phases:

Phase One. Test case design and confirmation of correct application behavior using three OS/Browser combinations.
Phase Two. Execution of failed tests and selected tests against remaining three browser/OS combinations.
Phase Three. Follow-up execution of selected tests against browser/OS combinations as necessary.

Operating System	Netscape	Internet Explorer	AOL
NT	4.72	5.01	No
98	4.72	5.01	No
95	4.72	5.01	No
Mac	No	No	No

The above diagram shows the various platforms that QA will test on. The browsers to be tested are the latest IE and NS browsers with 128 encryption. There are no plans to test 56, 40, and 32 encrypted browsers. There are no plans to test against other versions of IE and NS or to test the AOL browser. 128 encryption on the Mac will not be tested.

The purpose of this testing effort is to ensure that BPM maintains current functionality in a 128 encryption production environment. Backward compatibility is not being addressed, with other encryption browsers (56, 42 and 30). This testing effort will also focus on capturing any hard coded hyperlinks that may be in the BPM code. QA will be attempting to capture most of the hard coded hyperlinks, but because there are many functional combinations to be exercised, not all hard coded hyperlinks may be found.

Compatibility issues and comparison analysis of Netscape versus Internet Explorer will not be done. However, in the process of testing BPM, defects will be written that may address compatibility issues. QA Engineers will not specifically test for this or write test cases for these compatibility issues, but will be aware of any current problems. A document was distributed that will be reviewed for information, but not specifically addressed. This project is strictly focused on 128 encryption and the functionality of BPM in this environment, against IE and NS 128 encryption browsers.

What are hard coded hyperlinks and how does this relate to the 128 encryption project? In a non-secured environment hyperlinks are identified as http:// and in a secured environment https://. Code may exist within the BPM application that is has a hard coded URL, for example, http:// ……/RFI, when executing a BPM RFI function. This code will not work in a secured environment and therefore must be replaced with a variable that can call http: or https: hyperlinks, depending whether the customer is signing on to a secured or non-secured environment. QA will be executing these hyperlinks, while exercising the functionality of the BPM application. The hyperlink will be transparent to the QA Engineer, but the functionality will either work or it will not work and this is the reason for a BPM full regression test.

A full regression will be done on the BPM application. There are 3 different QA Engineers working on different features of the application. Application features will be regression tested across the various operating systems.

QA Engineer

Week 1

Week 2

Week 3

Aaron

NT w/IE 5.01

98 w/IE 5.01

95 w/NS 4.72

David

95 w/IE 5.01

NT w/IE 5.01

98 w/NS 4.72

Chris

98 w/IE 5.01

95 w/IE 5.01

NT w/NS 4.72

The above configuration will allow us to execute 100% of BPM test cases against IE and NS and 100% against NT, 95 and 98. Approximately 75% of our customers are using IE and 25% NS. Our primary focus will be IE. No other ClientCompanyName products are being addressed for 128 encryption, only BPM

At the completion of functional testing, BIDDERCOMPANYNAME and ClientCompanyName will review all outstanding defects. The ClientCompanyName Director of QA will chair this meeting. If required, additional rounds of functional testing will be done.

Defect Tracking and Reports

All defects found will be tracked in the Siebel defect control system. Defect Reports will be generated and presented to the ClientCompanyName Program Manager. BIDDERCOMPANYNAME and ClientCompanyName will review these defects jointly, as required. Severity 1 and 2 defects, with a Priority 1 and 2, will be fixed by ClientCompanyName in a timely manner and retested in the ClientCompanyName test environment.

Team Member Roles and Responsibilities

· Frank (ClientCompanyName Director of QA) - Responsible for the coordination of the testing effort within the ClientCompanyName QA test environment. He will work with the BIDDERCOMPANYNAME Team Lead and the ClientCompanyName Program Manager to ensure that the Project Team delivers a quality product.

· Josh (ClientCompanyName Product Marketing Manager) – A representative from the ClientCompanyName Marketing group will be involved to assist in the management of the project.

· Scott (ClientCompanyName Development Engineer) – A ClientCompanyName Development Engineer will be available to assist BIDDERCOMPANYNAME with any technical questions.

· Bill (ClientCompanyName QA Engineer and Test Coordinator) – Will work with the BIDDERCOMPANYNAME QA Team and will assist with the coordination of all testing activities.

· BIDDERCOMPANYNAME (QA Engineer(s)) - Will work with the ClientCompanyName QA Test Coordinator in executing the funcitional components of this test plan. ClientCompanyName and BIDDERCOMPANYNAME QA Engineers will execute all functional test cases, log all test results and compile reports. Logging the success or failure of testing efforts will be a joint effort between the ClientCompanyName Test Coordinator and BIDDERCOMPANYNAME testers.

Assumptions, Prerequisites and Risks

12.1 Assumptions

Adequate specifications or subject matter expertise will be available to assist reverse engineering in a timely fashion.

Any defects in the current release of the ClientCompanyName product will not impede understanding of functionality and reverse engineering of the product for test purposes.

A stable test environment is available for test, including essential test data.

Resolution of ambiguities can be resolved quickly

12.2 Prerequisites

Test environment with test data and Bidding product installed prior to User Acceptance testing.

Test cases approved and selected for User Acceptance testing.

Test team in place by 4/7/00.

12.3 Risks

Testing will be very restricted in terms of the functionality exercised. Field edits, database updates and cross-application functionality will not be explicitly exercised. Workflow functionality may not guarantee that every valid code or indicator is exercised if there is no impact on workflow functionality.

128-bit security testing will be exercised by exercising the functional test cases. However there is a risk of defects caused by "hard coding" of page identifiers used for hyperlinks. These links may not be visible from the screens tested during functional testing and thus may not be detected in test.

No retest of 32, 40 or 56-bit secure browsers will be performed. In addition, other combinations of browsers not listed in the "Test Platform" section of this plan will not be tested. This includes testing for AOL, IMAC and dial-up configurations.

Appendix A. Test Completion Schedule

Test Suites

Rev. Eng.

T.C. Des.

W/T - Rev.

Setup

First Pass Test

Cert.

Test Cases

Vars.

Quest.

Who - BIDDERCOMPANYNAME

Who - Guru

Notes

Admin.

Company Profile

2-May

4-May

5-May

A. Carvell

Photos Admin

28-Apr

A. Carvell

Project Image

27-Apr

C. Saul

Project Profile

2-May

3-May

4-May

D. Fuller

Project Update Admin

8-May

9-May

D. Fuller

Deactivate Users

8-May

9-May

D. Fuller

User Admin

3-May

4-May

5-May

D. Fuller

Area Reference

28-Apr

C. Saul

Distribution Admin

28-Apr

C. Saul

Role Assignments

28-Apr

C. Saul

Cert. Payroll Admin

3-May

5-May

8-May

C. Saul

Cert Payroll Rates

9-May

10 M

11 M.

C. Saul

Global - Test together

{ Login

27-Mar

28-Mar

C. Saul

{ Projects

C. Saul

{ Today

C. Saul

{ Logout

C. Saul

Server Attachments

11-Apr

13-Apr

14-Apr

D. Fuller

*SPEC

Design Apps

Daily Contract Report

17-Apr

18-Apr

A. Carvell

Daily Traffic Report

18-Apr

19-Apr

A. Carvell

Non-Conformance

22-Mar

23-Mar

24-Mar

D. Fuller

*SPEC

Field Observation

23-Mar

24-Mar

A. Carvell

Permit Tracking

25-Apr

26-Apr

A. Carvell

*SPEC

Punch List

24-Apr

25-Apr

A. Carvell

*SPEC

RFI

27-Mar

29-Mar

30-Mar

31-Mar

A. Carvell

*SPEC

Submittals

27-Mar

29-Mar

31-Mar

3-Apr

D. Fuller

*SPEC

Supplemental Instr.

4-Apr

6-Apr

7-Apr

10-Apr

A. Carvell

*SPEC - may need to split

Cost Apps

Owner Issues

30-Mar

31. M

C. Saul

*SPEC

Field Orders

3-Apr

4-Apr

5-Apr

C. Saul

*SPEC

Potential Change Orders

5-Apr

6-Apr

7-Apr

C. Saul

*SPEC

Owner Change Orders

10-Apr

11-Apr

12-Apr

C. Saul

*SPEC

Project Apps

Announcements

12-Apr

13-Apr

C. Saul

Forms

17-Apr

18-Apr

D. Fuller

Meeting Minutes

5-Apr

6-Apr

7-Apr

D. Fuller

*SPEC

Memo

18-Apr

19-Apr

D. Fuller

*SPEC

ROC

19-Apr

20-Apr

D. Fuller

References

20-Apr

21-Apr

D. Fuller

Schedules

24-Apr

25-Apr

D. Fuller

Daily Inspector Report

25-Apr

26-Apr

D. Fuller

Direct Transmittal

26-Apr

27-Apr

D. Fuller

Compliance

Certified Payroll/Report

11-Apr

13-Apr

14-Apr

A. Carvell

Proj. Info. - Test Tog.

{ Team Directory

27-Apr

28-Apr

D. Fuller

{ Photo Album

D. Fuller

{ Project Summary

D. Fuller

{ Project Updates

D. Fuller

{ Specifications

D. Fuller

{ Weather

D. Fuller

Settings - Test Tog.

User Profile

26-Apr

A. Carvell

Change Password

27-Apr

28-Apr

A. Carvell

User Preferences

27-Apr

A. Carvell

Utilities

New Project Wizard

19-Apr

20-Apr

21-Apr

A. Carvell

Project Applications

18-Apr

19-Apr

C. Saul

Application Admin

13-Apr

14-Apr

C. Saul

Minute Admin

17-Apr

C. Saul

Submittal Admin

17-Apr

18-Apr

C. Saul

Actor Assignments

13-Apr

14-Apr

C. Saul

Default Values

19-Apr

C. Saul

Permit Types

20-Apr

C. Saul

Header/Footer

21-Apr

C. Saul

***?Application Styler

24-Apr

C. Saul

These features are only accessible to ClientCompanyName Engineering.

***?Default Styler

24-Apr

25-Apr

C. Saul

***?BPM Admin

25-Apr

26-Apr

Operating Systems	Hardware Model	Manufacture	Modem
Windows 95	P5-350	Gateway	Dial-up-56K
Windows 98	Celeron 466	HP	Intranet
NT	P5-300	Gateway	Intranet

Product	Version	Manufacture	Description
Internet Explorer HP Celeron 466-98	5.0	MS	Internet Browser
Netscape HP Celeron 466-98	4.7	Netscape	Internet Browser

QA Engineer	Week 1	Week 2	Week 3
Aaron	NT w/IE 5.01	98 w/IE 5.01	95 w/NS 4.72
David	95 w/IE 5.01	NT w/IE 5.01	98 w/NS 4.72
Chris	98 w/IE 5.01	95 w/IE 5.01	NT w/NS 4.72