To run a test script from any intermediate step, use the €œRun from Step€ option to run a selected part of your test. This enables you to check a specific section of your AUT or to verify whether a certain part of the test script runs smoothly or not.

In the Expert View, you can use the €œRun from Step€ option to run your test from the selected step until the end of the action. If the action contains nested actions, QTP runs the nested actions for the defined number of iterations of the nested action. Also keep in mind that using Run from Step in this mode ignores any iteration.

In the Keyword View, you can use the €œRun from Step€ option to run your test from the selected step until the end of the test. Using €œRun from Step€ in this mode includes all iterations. The first iteration will run from the step you selected until the end of the test; all other remaining iterations will run from the beginning of the test.

If you only want to run single iteration of your test, select €œRun one iteration only€ from the Run pane in the Test Settings dialog box.

You can insert a breakpoint if you wish to run your test until a specific point within the test script (and not to the end of the action or test). The test will then run from the selected step or action until the breakpoint.

To run an entire action, or run a test or action from a selected step:

1. Make sure AUT is in a state matching the action or step you want to run.
2. Select the action or step where you want to start running the test. Make sure that the step or action you choose is not dependent on previous steps, such as a retrieved value or a parameter defined in a previous step.
3. Select Automation > Run from Step or Run Current Action, or right-click and select Run from Step. The Run dialog box opens.
4. In the Run dialog box, choose where to save the run session results, and define any input parameters you want to use.
5. Click OK. The Run dialog box closes and the run session starts

Why Should you  Build?

It is observed that any software project can benefit from a standardized, formal build process. Delivering quality software also mean producing quality deliverables, manually producing those deliverables will definitely be prone to human errors.

Defining the build process is something that should be done very early on in the development effort. In many development projects, daily builds are used to provide a way of ensuring that the project is not missing its objective.

Daily builds help developers ensure that the project can deliver something at any time during the lifespan of the sprint. When developers are working on different parts of the application, breakages can occur quite often when those separate efforts are checked into your source code version control system. These sorts of breakages are often only discovered when you attempt to build the entire project from the latest version of the source code.

There are many noticeable benefits in doing daily builds:

• Minimize risk: find out about incompatibilities when they occur in real time. This makes it easier to manage these incompatibilities and easier to fix ASAP.
• Improved quality: The product is always in a usable state, as each build or latest state of software is always sanitized.
• Speed up bug fixing: It is always easier and quicker to fix bugs when they appear rather than later, as it is tested on daily basis.

When to Build?

Noon Build – Have the developers check in any changes they have completed before noon, then who ever is responsible for the build process can kick off the build before they leave for lunch. Building during the day means that issues that arise from testing of the build can be addressed as soon as possible, while they are still fresh in the developer’s minds. If the daily build is passed before the team finishes for the day, then no time need be wasted fixing it the next, the team is clear on the build status.

If the build takes too long then building during the night is more practical.

Automate Build€€œ As it is obvious, Humans are not very good at repetitive tasks, often performing steps out of order, or missing steps altogether. An automated build process overcomes this by being able run the build process in a repeatable manner.

Build all, no partial business: “The Daily Build” should build the entire project, compiling every line of code in the project and produce every executable, dll or library.

If the build fails, fix it and then build again – The longer the issues that cause a build to fail are left unresolved, the harder they become to resolve later. Fix the issues as soon as possible, if the build is not passed for the day, a mail should be sent to the management notifying the issues.

Label successful builds in the version control system – This make it easier to revert to a “known good build” when things go horribly wrong or when some weird impossible to find bugs are found.

Archive the binaries for each build – This will help you find those weird or impossible to reproduce bugs. You can simply test the applications to find out when the bug began appearing, this gives the starting point of code changes that may be the main cause of the problem.

Lets talk about Root Cause Analysis and Defect Prevention and how they can be used for project betterment.

Firstly, Software Quality process and defect prevention are essentially one and the same. First, quality has to be established in the culture of the organization including not just the testers, but developers, project managers and product owners.

Second, defect prevention, sounds reassuring but practically is somewhat a myth. There will always be defects, but changes to how the development process can make discovering the problems earlier:

• Pragmatic use of unit tests
• A buddy system for checking in code through code reviews
• Constant code re-factoring
• Continuous integration
• Static code analysis (source code, byte code, path coverage, memory leaks, code coverage)
• Creation of development standards and guidelines in the team

Once the code gets developed, the testers shall use numerous techniques towards testing the deliverable. Using exploratory testing in agile will add more value, as it is nothing but simultaneous test design, documentation (to some extent) and execution. This has the advantage over traditional scripted tests in that it does not constantly recycle the same test data over and over (in most applications, a single set of test data will find a bug only very rarely), let the tester utilize his/her experience and intelligence in determining problems, and counteracts a large part of a testers intentional blindness.

Root cause analysis not really the domain of a tester in the strict sense. Since the cause of bugs is ultimately in the code, it is up to those with the most knowledge of the code, the development team, to identify the root cause of problems. However, it is possible to determine some aspects of the root cause externally by testers:

• Analyzing the code changes made for a series of fixes for things like:
  • common modules €€œ lets re-factor?
  • common developer origin €€œ Greenhorn, experienced, expert? Need for some additional training?
  • common defect types
  • Base on historical testing data

Once the quality infrastructure is in place, the net gets tightened earlier around problems and root cause analysis becomes less of a priority and/or need. When the space shuttle was originally being created, every bug went through root cause analysis and that pattern was then searched for through the whole code base. With more and more agile techniques and today’s tools those glory days are thankfully behind us to most extent.

For any testing activity to begin with, requirement analysis and understandng is of prime importance for effective and efficient testing. Below are few points of how a test team should work towards gathering requirements and carry on with their testing in an agile development model.

Pre-iteration:

The requirements are analyzed with product owners/BAs, acceptance criteria for the iterations are chalked out. The test team should verify the requirements and raise queries/concerns (before the iteration kick off)

Requirement analysis:

- Test team will look at requirements for “Clarity” and “Testability”

- Test team should really need to look at the requirement/stories upfront to figure out that requirement are “unambiguous” and “testable”.

- Giving feedback early on will help test team and also makes easy for developers to understand requirement

- Each requirement/user story should have few mandatory fields like

€œDescription€

€œGoal€ – task that they need to perform

€œWhy€ – an explanation of €œwhy€ it is needed and the associated value

€œPriority€

- Test team will have a discussion of all the requirements planned for the iteration with BA (ideally with dev lead present); the meeting will be called as €œRequirement Analysis Meeting€

- Requirement should be small enough to make sense in the testing context (This can be achieved by brainstorming with BAs/Developers/tester)

- Test team will scan through the requirement/story to see what needs to be done to test the requirement/story, like

- Finding out hidden requirements

- Environment required

- Test data

- Dependency on other requirement.

- Each User Story must be defined in terms of business value.

- Each User Story should not span across sprints.

If the above is clear, it will assist test team to prioritize the testing based on backlogs and plan for the particular iteration

Within iteration:

Once the above two is achieved to satisfactory level,

- Acceptance test cases shall be written by test team

- Acceptance tests should not be covering too many corner case scenarios

- Acceptance test cases traverse only happy path

- These acceptance test cases are functional cases.

- All corner cases shall be covered in exploratory testing

- The test team will deliver the acceptance test cases for dev team

- Dev team will run the acceptance test cases as soon as they develop to ensure that sanity of the build is intact.

- Once the acceptance test case execution is complete and passes as expected, the build is ready for the test team

- Test team will test (acceptance + exploratory + context driven)

- Once test team Okays the iteration, it shall be released to customers for further process

- In exploratory, its test-first, document-later approach, reporting high priority defects to dev team as soon as possible is more important than documenting the test cases.

- More time for exploratory testing (or context driven testing)

- About exploratory – Exploratory testing is a not at all about executing a pre defined testing scenarios/cases, it is a testing methodology that is more of exploring the software beyond test cases but at the same time making sure the focus is around some specific requirements (user stories).

System testing in an agile development environment is a challenge, but the benefits to having good testing practice always add value to the overall project.

Few practices that can ensure agile testing success can be:

1. Test Early – The key to agile testing is iteration: Develop -> Test the developed code -> Develop again -> Test the latest developed code, the key here is starting the testing activity early. Here testing should start at the very beginning of the development cycle rather than after the first few sprints.

2. Test Often – A good agile test process emphasizes on frequency of testing. Here the objective should be to find as much as defects early in the testing cycle. The longer the defects wait in the code, the harder and more expensive they will be to fix.

3. Code Refactoring/Regression €€œ Stop every few weeks (at regular intervals) to focus on overall system stability. Fix bugs, refactor old code, and run extensive regression testing suite to make sure that no bugs slipped during the ongoing testing activities. This cycle should be repeated periodically

4. Test from a Customer (Business) Point of View – As with any development process, it’s critical that the testers and developers know the customer’s point of view. That means having good business scenarios that are derived from the customer, and then sharing those scenarios with the development team as well as the testing team.

5. Separate Testing from Development – This is often difficult for smaller teams, but testers should be always independent from development team. Keeping testers separate means they can develop true testing expertise while focusing on finding bugs. This also helps testers in having a perspective that is different from development team that will eventually help in finding bugs that are out of developer’s imagination.

6. Effective Communication €€œ Having good communication between the testers, developers, and product guys is a key essential to a solid agile process. Even though testers should be separate from developers, they should work closely together to get the most from testing activity.

7. Automate what you can – The best agile teams automate as much of their manual testing load as they can. Repeatedly testing the same test case over and over is a waste of time and money. Locating new bugs is far more valuable for any testing team than running same test cases again and again.

As the key to Agile development is speed, the key to testing in an Agile development project is also speed – creating test plans and test cases quickly, producing test results quickly, structuring your tests so that they are easy to update, and a test suite that withstands minor UI changes and/or needs little effort to maintain when it does break at some point of time.

There are number of problems that may be encountered in trying to automate testing. Having some idea of the type of problems that encounter should help in implementing an effective automation regime.

Few common problems are described below.

1. Unrealistic expectations. Generally there is a tendency to be optimistic/have high expectation about what can be achieved by a new test tool. It is human nature to hope that this new test solution will at last solve all of the problems we are currently experiencing. Vendors usually emphasize the benefits and successes, and may play down the amount of effort needed to achieve the desired benefits. If management expectations are unrealistic, then no matter how well the tool is implemented from a technical point of view, it will not meet expectations.

2. Expectation that automated tests will find a lot of new defects. A test might more likely find a defect the first time it is run. If a test has already run and passed, running the same test again is much less likely to find a new defect (unless the test is exercising code that has been changed or could be affected by a change made in a different part of the software, or is being run in a different environment).

Test execution tools are ‘record – replay’ tools, i.e. regression testing tools. Their use is in repeating tests that have already run. This is a very useful thing to do, but it is not likely to find a large number of new defects, particularly when run in the same hardware and software environment as before. Knowing that a set of tests has passed again gives confidence that the software is still working as well as it was before, and that changes elsewhere have not had unforeseen effects.

3. Poor testing practice. If testing practice is poor, with poorly organized/designed tests, little or inconsistent documentation and tests that are not very effective at finding defects, automating those tests is not a good idea.

4. Maintenance of automated tests. When software is changed it is often necessary to update some, or even entire test suite, so they can be re-run successfully. This is particularly true for automated tests. Test maintenance effort is the biggest challenge and often reason to truncate many test automation initiatives. When it takes more effort to update the tests than it would take to re-run those tests manually, test automation will be stopped.

5. False sense of security. Just because a test suite runs without finding any defects, it does not mean that there are no defects in the software. The tests may be incomplete, or may contain defects themselves. If the expected outcomes are incorrect, automated tests will simply preserve those defective results.

6. Technical problems of tools. Commercial test execution tools are software products, sold by vendor companies, they are not immune from defects or problems of support. Interoperability of the tool with other software, either your own applications or third-party products, can be a serious problem. Many tools look ideal on paper, but simply fail to work in some environments.

In addition to technical problems with the tools themselves, we may experience technical problems with the software we are trying to test. If software is not designed and built with testability in mind, it can be very difficult to test, either manually or automatically. Trying to use tools to test such software will add complication which will only make test automation even more difficult.

7. Organizational issues. Automating testing is not a trivial exercise, and it needs to be well supported by management and implemented into the culture of the organization. Time must be allocated for choosing tools, for training, for experimenting and learning what works best, and for promoting tool use within the organization.

Test automation is an infrastructure issue, not just a project issue. In large organizations, test automation can rarely be justified on the basis of a single project, since the project will bear all of the start-up costs and teething problems and may reap little of the benefits. If the scope of test automation is only for one project, people will then be assigned to new projects, and the automation initiative will be lost.

Test automation can enable few testing tasks to be performed far more efficiently than could ever be done by testing manually and hence adding value to overall testing efforts.

Some of the benefits of automation tests/testing are listed below

1. Regression tests. Most suitable to run regression (for existing features) tests on the latest version of the application. This is perhaps the most obvious task, particularly in an environment where frequent bug fixes are performed hence leading to frequent feature modifications. The effort involved in performing a set of regression tests should be minimal. Given that the tests already exist and have been automated to run on an earlier version of the application, it should be possible to select the tests and initiate their execution with just a few minutes of manual effort.
2. Difficult manual tests. Perform tests which would be difficult or impossible to do manually. For instance, to perform a full-scale live test of an online system with 1000 users may be impossible, but the input from 1000 users can be simulated using automated tests. By having end users define tests that can be automatically replayed, user scenario tests can be run at any time even by technical people who do not understand the intricacies of the full business application.
3. Consistency and repeatability of tests. Tests that are repeated automatically will be repeated exactly every time without fail. This gives a level of consistency to the tests which is very difficult to achieve manually. The same tests can be executed on different hardware configurations, using different operating systems, or using different databases. This gives a consistency of cross-platform quality for multi-platform products which is virtually impossible to achieve with manual testing. The good automated testing practice can also insure consistent standards both in testing and in development.
4. Verifying invisible results. In manual testing, typical expected outcomes are the obvious things that are visible to the tester. However, there are cases where attributes that should be tested are not easy to verify manually. For example a graphical user interface (GUI) object may trigger some event that does not produce any immediate output or that is not visible to the tester. A test execution tool may be able to check that the event has been, triggered, which would not be possible to check without using a tool.
5. Reuse of tests. Tests that are reusable across many releases or test cycles are best to automate. The effort put into deciding what to test, designing the tests, and building the tests can be distributed over many executions of those tests. Tests which will be reused are worth spending time on to make sure they are reliable. This is also true of manual tests, but an automated test would be reused many more times than the same test repeated manually.
6. Earlier time to market. Once a set of tests has been automated, it can be repeated far more quickly than it would be manually, so the testing cycle time can be shorter.
7. Increased confidence. Knowing that an extensive set of automated tests has run successfully, there can be greater confidence that there won’t be any unpleasant surprises when the system is released (providing that the tests being run are good tests!). In short, more thorough testing can be achieved with less effort, giving increases in both quality and productivity.

We shall discuss about the challenges of automated tests/testing in the following blog!