ISTQB CTFL Syllabus 2011

The strength of deci sion table te sting is that it creates co mbinations of conditions that otherwi se might no t have been exercised during testing. It may be a pplied to all situations when the acti on of the softw are depends on several logical deci sions.

4.3.4State Tra nsition Testing (K 3)

A syste m may exhibit a different response de pending on current cond itions or previous history (its state). In this case, that aspect of the system can be sho wn with a sta te transition diagram. It allows the teste r to view the software in terms of its states, transitions between states, the inputs or e vents that trigg er state changes (transitions) and the actions wh ich may result from those transitions. The states of the system or object under test are separate, id entifiable an d finite in nu mber.

A state table shows the relationship between the states and inputs, a nd can highlight possible transitio ns that are in valid.

Tests can be designed to cover a typical sequence of states, to cover every state, to exercise every transitio n, to exercis e specific sequences of transitions or to test invalid transition s.

State transition testi ng is much used within th e embedde d software industry and technical automation in gener al. However, the techniq ue is also suitable for modeling a business object having specific states or testing s creen-dialo gue flows (e.g., for Internet applications or busine ss scenarios).

4.3.5Use Case Testing (K2)

Tests can be derived from use c ases. A use case describ es interactio ns between actors (users or systems), which pro duce a result of value to a system us er or the customer. Use cases may be described at the abstract level (business use case, technology-free, business pro cess level) or at the syst em level (sys tem use cas e on the sy stem functio nality level). Each use case has preconditions which need to be met for the u se case to work successfully. Each use case terminates with post conditions which are the observable results and final state of the system a fter the use case has been completed. A use case usually has a mainstream (i.e., most likely) scenario and alte rnative scenarios.

Use cases describe the “process flows” throu gh a system based on its actual likely use, so th e test cases d erived from u se cases are most usef ul in uncovering defects in the proces s flows duri ng real-world use of the system. Us e cases are very useful for designing acceptance tests with customer/user participation. The y also help uncover integ ration defects caused by the interaction and inte rference of different components, which individual component testing wou ld not see. Designin g test cases from use c ases may be combined w ith other sp ecification-b ased test techniques.

© Internationa l Software Testing Q ualifications Board

Terms

Code coverage, decision covera ge, statemen t coverage, structure-based testing

Background

Structur e-based or w hite-box testing is based on an identified structu e of the software or the system, as seen in t he following examples:

oCom ponent level: the structure of a softw are component, i.e., stat ements, dec isions, branches or e ven distinct paths

oInte gration level: the structure may be a call tree (a diagram in wh ich modules call other modules)

o System level: th e structure m ay be a me nu structure, business process or web page structure

In this s ection, three code-relate d structural test design te chniques for code coverage, based on stateme nts, branches and decisions, are disc ussed. For decision testing, a contro l flow diagram may be used to visu alize the alternatives for each decision.

4.4.1Statement Testing and Coverage (K4)

In comp onent testing , statement coverage is the assessm ent of the percentage of executable stateme nts that have been exercised by a te st case suite. The statem ent testing technique derives test cas es to execut e specific sta tements, normally to increase statem ent coverag e.

Statement coverage is determine d by the nu mber of exec utable statements covered by (desig ned or execu ted) test cas es divided b y the numb er of all executable state ments in the code under test.

4.4.2Decision Testing and Cove rage (K4)

Decision coverage, related to bra nch testing, is the asses sment of th e percentage of decision outcome s (e.g., the True and False options of an IF statement) that have been exercised by a test case suite. The deci sion testing technique de rives test cases to exec ute specific d ecision outc omes. Branches originate f rom decision points in th e code and s how the tra nsfer of control to differe nt location s in the code.

Decision coverage is determined by the number of all de cision outco mes covered by (designed or execute d) test cases divided by the number of all possible decision o utcomes in t he code under test.

Decision testing is a form of control flow testing as it follows a specific flow of cont rol through the decision points. Decision covera ge is stronge r than state ment coverage; 100% de cision coverage guarante es 100% st atement coverage, but n ot vice versa.

4.4.3Other Structure-b ased Tech niques (K 1)

There are stronger l evels of struc tural coverage beyond d ecision cov erage, for example, condition coverag e and multiple condition coverage.

The concept of coverage can also be applied at other test levels For example, at the integration level the percentage of modules, component s or classes that have be en exercised by a test case suite co uld be expressed as mod ule, compon ent or class coverage.

Tool sup port is usefu l for the stru ctural testin g of code.

© Internationa l Software Testing Q ualifications Board

Terms

Exploratory testing, (fault) attack

Background

Experien ce-based testing is where tests are derived from the tester’s skill and intu ition and th eir experien ce with similar applicatio ns and technologies. W hen used to augment sys tematic

techniques, these techniques ca n be useful i n identifying special tests not easily c aptured by formal techniques, especially when applied after more formal approaches. However, this technique m ay yield wid ely varying degrees of effectiveness, depending on the testers’ experienc e.

A commonly used ex perience-ba sed techniq ue is error g uessing. Ge nerally tester s anticipate defects based on experience. A structured a pproach to th e error gues sing techniq ue is to enumera te a list of possible defects and to d esign tests th at attack th ese defects. This systematic approach is called fault attack. These defect and failure lists can be built based o n experience, available defect and failure data, and from co mmon know ledge about why softwa re fails.

Exploratory testing is concurrent test design, test executi on, test logging and lear ning, based on a test char ter containin g test objectives, and ca rried out within time-boxes. It is an approach that is most us eful where th ere are few or inadequate specifications and severe time pressure, or in order to augment or complement other, more form al testing. It can serve as a check on the test proc ess, to help e nsure that t he most seri ous defects are found.

© Internationa l Software Testing Q ualifications Board

Terms

No specific terms.

Background

The choice of which test techniq ues to use d epends on a number of factors, including the type of system, regulatory standards, customer or co ntractual re quirements, level of risk, type of risk, test objective , document ation available, knowledg e of the testers, time and budget, de velopment life cycle, us e case models and previous experience with types of defect s found.

Some techniques ar e more applicable to certain situations and test levels; others are applicable to all test le vels.

When creating test cases, tester s generally u se a combin ation of test techniques including pro cess, rule and data-driven techniques t o ensure adequate coverage of the object under test.

References

4.1Craig, 2002, Hetzel, 1988, I EEE STD 829-1998

4.2Beiz er, 1990, C opeland, 200 4

4.3.1C opeland, 200 4, Myers, 1 979

4.3.2C opeland, 200 4, Myers, 1 979

4.3.3B eizer, 1990, Copeland, 2004

4.3.4B eizer, 1990, Copeland, 2004

4.3.5C opeland, 200 4

4.4.3 B eizer, 1990, Copeland, 2004

4.5Kaner, 2002

4.6Beiz er, 1990, C opeland, 200 4

© Internationa l Software Testing Q ualifications Board

Learning Obje ctives for Test Manageme nt

The obje ctives identify what you will be able to do followi ng the completion of each module.

5.1 Te st Organiz ation (K2)

LO-5.1.1 Recognize the impor tance of independent te sting (K1)

LO-5.1.2 Explain the benefits and drawba cks of indep endent testin g within an organization (K2) LO-5.1.3 Recognize the differ ent team members to be considered for the creation of a test team

(K1)

LO-5.1.4 Recall th e tasks of a typical test leader and t ester (K1)

5.2 Te st Planning and Estimation ( K3)

5.3 Te st Progres s Monito ring and C ontrol (K2)

5.4 Configuration Manage ment (K2)

LO-5.4.1 Summar ize how configuration m anagement supports test ing (K2)

5.5 Ris k and Te sting (K2)

© Internationa l Software Testing Q ualifications Board

© Internationa l Software Testing Q ualifications Board

Terms

Tester, test leader, t est manager

5.1.1Test Organization and Inde pendence (K2)

The effectiveness of finding defects by testin g and review s can be improved by us ing independent testers. Options for independence include th e following:

o No independent testers; dev elopers test their own code o Independent testers within th e development teams

o Independent test team or gro up within the organizatio n, reporting to project management or exec utive mana gement

o Independent testers from the business o ganization o r user comm unity

o Independent test specialists for specific test types such as usability testers, s ecurity testers or certification teste rs (who certify a software product ag ainst standa rds and regulations)

o Independent testers outsour ced or exter nal to the org anization

For larg e, complex or safety criti cal projects, it is usually best to have multiple lev els of testing, with some or all of the lev els done by independent testers. Development staff may participate in testing, especially at the lower levels, but their lack of objectivity often limits th eir effectiveness. The indepen dent testers may have the authority to require and define test processes a nd rules, but testers should take o n such process-related r oles only in the presence of a clear management mandate to do so.

The benefits of inde pendence in clude:

o Independent testers see oth er and differe nt defects, and are unbiased

oAn i ndependent tester can v erify assumptions people made during specificati on and implementation of the system

Drawba cks include:

o Isola tion from th e developme nt team (if tr eated as totally independent) o Developers may lose a sense of responsibility for qu ality

o Independent tes ters may be seen as a bottleneck or blamed for delays in rel ease

Testing tasks may b e done by pe ople in a specific testing role, or ma y be done by someone in another role, such a s a project m anager, quality manage , developer, business an d domain e xpert, infrastructure or IT operations.

5.1.2Tasks of the Test Leader an d Tester (K1)

In this s yllabus two test positions are covered , test leader and tester. The activities and tasks perform ed by people in these tw o roles depend on the project and pr oduct context, the people in the roles, an d the organization.

Sometim es the test leader is called a test manager or test coordinator. The role of the test leader may be performed by a project m anager, a development manager, a quality assu rance mana ger or the manager of a test group. In l arger projects two positions may exist: test leader and test manager. Typically the test leade r plans, mo nitors and co ntrols the testing activiti es and tasks as defined in Section 1.4.

© Internationa l Software Testing Q ualifications Board

o Contribute the te sting perspe ctive to othe r project activities, such as integratio n planning

oPlan the tests – considering the context and understa nding the test objectives and risks – inclu ding selecti ng test appro aches, esti mating the ti me, effort and cost of testing, acquiri ng

resources, defining test levels, cycles, an d planning incident management

oInitia te the specification, pre paration, imp lementation and execution of tests, monitor the test results and check the exit criteria

oAdapt planning b ased on test results and progress (sometimes d ocumented in status rep orts) and take any action necessary to compensate for problems

o Set up adequate configuratio n management of testw are for trace ability

oIntroduce suitable metrics fo r measuring test progress and evalua ting the quality of the te sting and the product

o Decide what sho uld be auto mated, to what degree, and how

Typical tester tasks may include:

o Review and cont ribute to test plans

o Analyze, review and assess user requirements, specifications an d models for testability o Cre ate test specifications

oSet up the test environment (often coordinating with s ystem administration and network management)

o Prep are and acq uire test data

oImplement tests on all test levels, execute and log th e tests, eval uate the resu lts and docu ment the deviations fr om expected results

o Use test adminis tration or m anagement tools and test monitoring tools as required o Auto mate tests (may be sup ported by a d eveloper or a test autom ation exper t)

o Measure perform ance of co mponents and systems (if applicable) o Review tests de veloped by others

People who work on test analysi s, test desig n, specific test types or t est automati on may be specialis ts in these roles. Depending on the test level and the risks related to the product and the project, different people may tak e over the role of tester, keeping som e degree of independence. Typically testers at t he component and integration level w ould be developers, testers at the accepta nce test level would be business experts and users, and teste rs for operational accep tance testing w ould be operators.

© Internationa l Software Testing Q ualifications Board

Terms

Test app roach, test strategy

5.2.1Test Pla nning (K2)

This section covers the purpose of test planning within de velopment and implementation proje cts, and for maintenance activities. Planning may be documented in a master test pla n and in sep arate test plans for test levels such as system testing and acceptance testin g. The outli ne of a testplanning document is covered by the ‘Standa rd for Software Test Doc umentation’ (IEEE Std 8 291998).

Planning is influenced by the test policy of th e organizatio n, the scop e of testing, objectives, risks, constrai nts, criticality , testability and the availability of resources. As the project a nd test plann ing progress , more information becomes available and more detail can b e included in the plan.

Test pla nning is a continuous act ivity and is p erformed in all life cycle processes a nd activities. Feedbac k from test activities is used to recog nize changing risks so t hat planning can be adjusted.

5.2.2Test Pla nning Activities (K3 )

Test pla nning activities for an entire system o r part of a s ystem may i nclude: o Dete rmining the scope and risks and ide ntifying the objectives of testing

oDefining the overall approac h of testing, including the definition o f the test lev els and entry and exit criteria

o Inte grating and c oordinating the testing activities into the software life cycle a ctivities (acquisition, supply, development, operation and maintenance)

o Making decisions about what to test, wha t roles will perform the t est activities , how the tes t

oSelecting metric s for monitoring and controlling test preparation and executio n, defect resolution and risk issues

oSetting the level of detail for test procedures in order to provide enough inform ation to sup port reproducible tes t preparation and execution

5.2.3Entry Criteria (K2)

Entry criteria define when to start testing suc h as at the beginning of a test level or when a set of tests is ready for exe cution.

Typically entry criteria may cover the following: o Test environmen t availability and readine ss o Test tool readine ss in the tes t environment o Testable code a vailability

o Test data availa bility

5.2.4Exit Crit eria (K2)

Exit crite ria define when to stop testing such as at the en d of a test level or when a set of tests has achieve d specific goal.

© Internationa l Software Testing Q ualifications Board

5.2.5Test Estimation (K 2)

Two app roaches for the estimati on of test effort are:

oThe metrics-based approach: estimating the testing e ffort based on metrics of former or similar proj ects or base d on typical values

oThe expert-based approach: estimating the tasks bas ed on estimates made by the owner of the tasks or by experts

Once th e test effort is estimated, resources can be identified and a schedule can be drawn up.

The testing effort may depend on a number o f factors, inc luding:

oCharacteristics o f the product: the quality of the specification and other information used for test models (i.e., the test basis), the size of th e product, t he complexity of the prob lem domain, the

requ irements for reliability an d security, a nd the requirements for documentation

oCharacteristics o f the development proce ss: the stability of the organization, tools used, te st process, skills of the people involved, and time pressure

o The outcome of testing: the number of d efects and the amount of rework requ ired

5.2.6Test Stra tegy, Test Approa ch (K2)

The test approach is the implem entation of th e test strategy for a specific project. The test app roach is defined and refined in the test plans and test designs. It typically in cludes the d ecisions ma de based o n the (test) project’s goal and risk as sessment. It is the starting point for planning the test process, for selectin g the test design techniques and test types to be applied, and for defining the entry and exit criteria .

The sele cted approach depends on the conte xt and may consider ris ks, hazards and safety, available resources and skills, the technolog y, the nature of the system (e.g., custom built vs. COTS), test objectives, and regulations.

Typical approaches include:

oAnalytical appro aches, such as risk-based testing where testing i s directed to areas of gre atest risk

o Model-based approaches, su ch as stochastic testing using statistical information about failure rate s (such as re liability grow th models) or usage (such as operat ional profile s)

o Met hodical appr oaches, suc h as failure-b ased (includ ing error guessing and fault attacks), exp erience-base d, checklist-based, and quality chara cteristic-bas ed

oProc essor standard-compli ant approac hes, such as those specified by indus try-specific standards or the various agile methodologies

oDynamic and heuristic approaches, such as explorat ory testing where testing is more rea ctive to eve nts than pre-planned, an d where execution and evaluation ar e concurrent tasks

oConsultative app roaches, such as those in which test coverage is driven primarily by the a dvice and guidance of technology and/or business domain experts outside the test team

oRegression-averse approach es, such as those that in clude reuse of existing test material, extensive automation of functional regres sion tests, a nd standard test suites

Different approaches may be co mbined, for e xample, a risk-based dynamic appro ach.

© Internationa l Software Testing Q ualifications Board