Risk-based testing

1.

RISK-BASED TESTING
HELEN VOROBEI
OCTOBER, 2017
CONFIDENTIAL
1

2.

INTRODUCTION
Helen Vorobei
Quality Architect,
Testing Competency Center Expert
10 years in testing
E-mail: [email protected]
CONFIDENTIAL
2

3.

AGENDA
1
What is risk and risk-based testing?
2
How to identify risks
3
How to apply risks for regression testing
4
How to apply risks for compatibility testing
CONFIDENTIAL
3

4.

TESTING GOAL
To find all defects
Find the most critical issues early
Critical issues
Require more time for fixing
May require great changes in architecture
Required a lot of retesting
May block testing
May affect testing and project delivery
milestones
CONFIDENTIAL
4

5.

WHAT CAN HELP? – RISK ANALYSIS
Analyze what areas are risky
Plan testing them more and earlier
Reduce testing effort for areas
where there is less risk
CONFIDENTIAL
Complexity
New Feature
User Traffic
Many bugs
Customer
Impact
Integration
Size of Code
Change
Project
Schedule
People
Issues
5

6.

RISK-BASED TESTING: DEFINITION
Risk-based testing is a testing done for the product based on risks. Risk-based
testing uses risk to prioritize and emphasize the appropriate tests/testing types/
test configurations/.. during test execution.
Risk is the probability of occurrence of an undesirable outcome, e.g. issue/bug
Risk attributes:
• Impact - cost when a risk does occur
• Probability - chance that a risk will occur
CONFIDENTIAL
6

7.

HOW TO IDENTIFY RISKS
CONFIDENTIAL
7

8.

RISK IMPACT
We get 10 new features for testing. Which bugs will be most important?
System down
Crash
Something that users use frequently
Main business flow
Legal consequences
Everything what many people use and what impacts business – talk to PO/ BA /business
CONFIDENTIAL
8

9.

RISK PROBABILITY
Which features probably will have more bugs than others?
Features with ambiguous requirements
Features with a lot of issues in the past
Technological complex features
Logically complex features
Features that implemented by New /junior developers
Features that were developed in a rush
Features that use new technology for the team
Features with a lot of changes
What else?
CONFIDENTIAL
9

10.

DEFECT ANALYSIS
There is not much difference between application areas. Do they have the same risks ?
How to prioritize their testing?
Look at defects: In areas with a lot of defects, probably there are still more
A lot of defects may indicate high risks like:
- poor code quality
- development complexity
- a log of changes were done
- developed in rush
…
=> So areas with a lot of defects may need more testing then others
CONFIDENTIAL
10

11.

HOW TO APPLY RISKS
CONFIDENTIAL
11

12.

RISK-BASED APPROACH FOR REGRESSION TESTING
Input: You have 600 test cases and need 20 days to run regression
Problem: Can you complete regression in 10 days?
What to do:
• analyze how many regression defects are found before and their severity
• analyze defect distribution by product components
• analyze where changes were done, evaluate these changes size and complexity
• ask developers what can be affected by new changes
CONFIDENTIAL
12

13.

RISK-BASED APPROACH FOR REGRESSION TESTING
Can we reduce testing effort for these cases?
Definitely, Yes!
Looks No!
Need analyze deeply
CONFIDENTIAL
13

14.

RISK-BASED APPROACH FOR REGRESSION TESTING
Analyze defects distribution between application components and different testing types
4 the most risky areas with the greatest number of defects
-high probability to find more defects during new testing phase
CONFIDENTIAL
14

15.

RISK-BASED APPROACH FOR REGRESSION TESTING
Where new changes were done? What is their complexity? What is size
of changes? Do they impact other functionality?
High risk to new
defect appearing in
areas where a lot of
changes were done
and impact high
CONFIDENTIAL
15

16.

RISK-BASED APPROACH FOR REGRESSION TESTING
Combine risk analysis by defects and by new changes and impact
CONFIDENTIAL
16

17.

RISK-BASED APPROACH FOR REGRESSION TESTING
Plan less testing in areas where there is less risk and more testing in areas where there
is high risks
• But not exclude any component from testing at all
What we get in result - reduce regression by 40%!
CONFIDENTIAL
17

18.

RISK-BASED APPROACH FOR TEST CONFIGURATIONS
5-10 configurations are in scope. Do we need to repeat the same tests on all
configurations? – Often No!
1. Set up priority based on usage statistic
2. Analyze defects distribution between configurations
3. Distribute tests between configurations and testing types
CONFIDENTIAL
18

19.

RISK-BASED APPROACH FOR TEST CONFIGURATIONS
1. Set up priority based on user usage statistic:
Use Analytics to get % of user sessions for top configurations
CONFIDENTIAL
19

20.

RISK-BASED APPROACH FOR TEST CONFIGURATIONS
2. Analyze defects distribution between configurations:
Number of Prod defects and total number of defects by different configurations
Calculate %
CONFIDENTIAL
20

21.

RISK-BASED APPROACH FOR TEST CONFIGURATIONS
3. Distribute configurations between tests, testing types and testing phases
More and earlier test more risky configurations
Test new features on High priority configurations (for example, divide browsers
between testers)
Regression testing on medium priority and just smoke tests on low priority congs
CONFIDENTIAL
21

22.

WHERE ELSE RISK-BASED APPROACH CAN BE APPLIED?
Reorder development plan
Refocus testing activities
start with most complex and risky areas
allocate more time on risky activities, for example, requirement analysis, optimize
efforts on not-risky (check list instead of test cases for not-complex functionality)
Everywhere!
CONFIDENTIAL
22

23.

REAL STORY FROM ONE OF EPAM PROJECTS
WHAT WAS BEFORE
• Regression testing took 3 months by team of 10 testers
• Development continued in rush and regression scope was growing
WHAT WAS DONE
• Analyzed regression defects showed that their main reasons are
Not fully tested features during new feature testing
Missed integration between functions
• Team defined more strict DoD to ensure that testing complete
• Change approach for regression testing to scenario-based
WHAT GOT IN RESULT
• Regression testing took 3 weeks execution
• Product quality increased
CONFIDENTIAL
23

24.

HOW TO START WITH DEFECT ANALYSIS?
1.
Define what information is important for you: components, configurations, testing types,
testing phase, etc.
2.
Specify bug reporting rules and share within the team, explain why you need this
3.
Perform analysis on regular basis.
4.
Use Excel and pivot tables, chart and diagrams: export defects for some period with all
fields. Period can be defined by releases – last 1-2 releases, for example.
5.
Revise results, ensure that your optimization/ changes in test strategy/ regression testing/
configuration testing do not affect product quality
CONFIDENTIAL
24

25.

MORE DETAILS ABOUT DEFECT ANALYSIS
Example of bug report ->
- Component tracked in component field
- Regression/UAT/automation/ new feature by labels
It is important to agree the rules what and how
to fill additionally to mandatory fields in bug
reports
CONFIDENTIAL
25

26.

MORE DETAILS ABOUT DEFECT ANALYSIS
Download all defects for some period with all
fields into Excel
Example from JIRA
CONFIDENTIAL
26

27.

MORE DETAILS ABOUT DEFECT ANALYSIS
In Excel: Insert -> Pivot
Table
In PivotTable Fields:
-
select Priority to
Columns
-
Select Labels to Rows
-
Select Key in Values
MS tutorial link
CONFIDENTIAL
27

28.

HOME TASK
1.
Analyze what areas are risky ( or NOT risky) for your application
2.
Propose how testing can be optimized or improved based on result analysis
CONFIDENTIAL
28