Re Run Failed Tests automatically |TestNG

We are using TestNG framework to manage and execute UI Tests. UI Tests might failed due to different reasons like connection timed out/browser hang/Network Problem...etc. This kind of issues is not related to Application under Test. In this case, we should re run the tests once again.

TestNG is providing default feature called Retry Analyzer which will re execute test case in case of failure.  TestNG Provides interface called IRetryAnalyzer with method called Retry. We need to override this method as per our requirement.

Example:

1| Assume you have a class file with 10 Methods [@Test]

2| Lets say one got failed due to Network issue

3| The class file, RetryAnalyzer will make the failed test-case to re-run with desired count; But, it display errors for all the test failures during re-run. [Explained in this post]

4| you are ought to add a Listener class, RetryTestListener; it will print the failure only once.

5| If the test got passed in the final count, failure exception won't be thrown in the console.

Test Class

@BeforeTest

public void setUp()

 {

System.out.println("This is Setup Method");

 }

@Test(retryAnalyzer=RetryAnalyzer.class)

public void test01() throws Exception

 {

        // Assert.fail();

 Assert.assertEquals("Failed", "Passed");

 }

RetryAnalyzer.java

import org.apache.log4j.Logger;

import org.apache.log4j.PropertyConfigurator;

import org.testng.IRetryAnalyzer;

import org.testng.ITestResult;

public class RetryAnalyzer implements IRetryAnalyzer  {

private int count = 0;

private int maxCount = 4; // set your count to re-run test

protected Logger log;

private static Logger testbaseLog;

static {

    PropertyConfigurator.configure("test-config/log4j.properties");

    testbaseLog = Logger.getLogger("TestclassName");

}

public RetryAnalyzer()

{

    testbaseLog.trace( " ModeledRetryAnalyzer constructor " + this.getClass().getName() );

    log = Logger.getLogger("TestclassName");

}

@Override

public boolean retry(ITestResult result) {

    testbaseLog.trace("running retry logic for  '"

            + result.getName()

            + "' on class " + this.getClass().getName() );

        if(count < maxCount) {                    

                count++;                                   

                return true;

        }

        return false;

}

}

The Test Class is having one test method with RetryAnalyzer class which will invoke RetryAnalyzer  whenever the test fails and RetryAnalyzer class re executes the same test method automatically.

Soft Assertions with TestNG

All of us writes automated tests and do the assertions as per our test cases. Each test case might have more than one or more assertions. Assume you have automated one test case and having more than one assertion. We all know that if first assertion in the test case fails then test case will be failed and do not validate remaining assertions in the test case.

For example, you automated one test case which validates UI Verification in the Webpage and DB Verification in the back end system...etc. Now you wanted your test case to validate both UI and DB Verification even any one of the verification is failed. It is not possible with hard assertion as test case will fail immediately when assertion condition fails.

TestNG support the following assertions.

Hard Assertions:

Test immediately fail and stop executing the moment a failure occurs in assertion. You may want to use hard assertion in case of Pre Condition of test case fails and no point of executing the test case in further.

Soft Assertion:

Tests don’t stop running even if assertion condition fails, but the test itself marked as a failed test to indicate right result. This is useful if you are doing multiple validations like UI Page Multiple elements Verifications, DB Verification...etc and you wanted to assert DB even any one of the UI Assertion fails and fail the test case once all the validations are complete in case of failures other Pass the test case.

Example:

package automation.tests;

import org.testng.asserts.Assertion;

import org.testng.asserts.SoftAssert;

public class Sample {

  private Assertion hardAssert = new Assertion();

  private SoftAssert softAssert = new SoftAssert();

}

 @Test

public void testForHardAssert() {

  hardAssert.assertTrue(false);

}

 @Test

public void testForSoftAssertWithNoFailure() {

  softAssert.assertTrue(false);  

}

 @Test

public void testForSoftAssertionFailure() {

  softAssert.assertTrue(false);

  softAssert.assertEquals(1, 2);

  softAssert.assertAll();

}

}

If you look at the test case (testForSoftAssertionFailure), the softAssert.assertAll() does the trick instead of writing your owned custom logic. This method collates all the failures and decides whether to fail the test or not. So instead of writing custom logic, the TestNG library itself offers the facility to perform soft assertions in your test.