Releasing a product?

Today is the day to release the product, the team worked hard spending countless hours engineering the new features. But are the things ready?

Do we know the product that is being released is ready to be shipped? Who calls out that we are ready? And what all we did prior to release?

Improving the product and means improving the quality of product and also improving the release process of the product.

Having a improved release process will produce best quality of engineering and thus reduces the headaches for customers. In current world, the releases are fast, which is great however, releasing fast does not mean releasing with quality. Important areas of product could be missed.

Team members with various roles and skill set have various perceptions about the features and usage of features in the product. The product might be tested thoroughly, however, considering the views of various members in the team before release is important. To make the point simple, there are lot more things to consider before releasing the product than just developing, testing and releasing.

Few things to keep in mind as you are coming closer to a release.

• Have we communicated everything? Any ambiguity? Did we follow best practices and processes? 
• Testing criteria should be detailed, specified and communicated that what and how we tested and what and why we did not test. 
• Are the views and perceptions related usability and user experience has been taken into consideration?
• Any external factors such as events, holidays been considered?
• Has the Technical support team prepared and had experienced the new features.

Customers and Engineering both should be happy to make a successful release.

Monitoring application logs

Logs have crucial information about the application behavior. Messages in the logs have tags and are shown under FATAL, ERROR, WARNING and INFO sections. And these messages could be related to syntax, improper usage, unhandled exceptions, data issues and processing. Most often the FATAL and ERROR messages get enough attention by a Software tester, but some of the most important messages that are listed as WARNING and INFO messages are ignored. The probable reason is that there are some hundreds of different types of WARNING and INFO messages, and most of them are just informative and not considered as important, and developer or tester gets overwhelmed to check all of them. But there are certain WARNING and INFO messages that ought to be checked as they are as equally important as FATAL / ERROR messages and indicate potential issues in the application code or logic or data.

Upon looking for the issues that one need to check in the log, there are many (in hundreds), but those messages are worth checking, every time a application is run. Apart from letting the user know the syntax mistakes in the application, the log file also provides vital information about the correctness of the logic / algorithm used in the application.

It might not practical be to check all the messages in application log files every time they were run, considering the huge volume of the log files, however there are certain messages and information that must be checked to make sure that there are no / minimal issues in the application behavior. Software testers must pay attention to these messages / information in the log and always must check them keeping in mind the consequences of ignoring them.

There are few nice tools which do all the work for you.

http://www.hootech.com/WinTail
https://www.sumologic.com

Some extremely powerful features are:
1. Filters that makes sure only lines with text of your choice are shown
2. Email notification whenever certain text is found
3. Monitoring many files at once
4. Highlight specific text (this enables you to highlight functional way points of the application)
5. Easy to see when a log file has been updated
6. Open the original log file in only one click

Validate Styling of Web Application using Telerik Test Studio

Apart from Functional, User Interface layout verifications, Test Studio can be used for validating the Styles of a web page which is under test.

I came across this feature, while evaluating an approach to test the web application which has multiple themes. Your automated tests should even run when the styling of your web page has changed. To ensure that styling did not impact any User Interface layout and caused functionality issues.

Let's see how to do it.

1. Start recording your web page.
2. Highlight the element that you want to validate for CSS.
3. In the DOM explorer, Select the element and click on Style in Verifications. (as shown in below image).

There are two Primary in the Styles that you need to choose depending on the web page under test. Basically, you should know, whether your application under test styling comes from a CSS file or from a web page itself.  
a. Computed Style - A inherited or cascading styles
b. Inline Style

Mostly, the modern web applications use CSS file from where the styles are used.

There are many options in Style, which lets you validate almost all variations of a web page styling.

You could also use regular expressions in case the web page styling changes dynamically.
For e.g. Themes.

4. Once you have selected to use a specific criteria, Select Add step. This will create a step in the Test.

In this way you can test the Style's of a web application.  

Short regression cycles in sprint

In agile process, each new functionality is tested in the sprint which helps the Engineering team to get feedback at the earliest and to make the process go smoothly, it is crucial for development team to provide a successful build to testing team.

QA will test the feature or functionality implemented in the sprint and log defects and the DEV team triages and fixes the defects. This all looks fine.

The problem or the challenge is:
The feature or the functionality would have been tested thoroughly however, integration points might not be covered. The integration points might be between products, components or features.
Some organisations have a different team for testing integration points between products. Though testing is not done in every sprint. Still the component, feature level integration are the responsibility of the sprint team.
In general, testing these integration points are only performed once or twice in the end of a release cycle and lot of defects are filed and fixed. A phase wherein a stressed release end cycle for the engineering team is observed.

Typically, a quick regression cycle at the end of sprint would make things look easy for the engineering team. However, it is difficult to achieve in sprints less than 3 weeks.

To make things simple, you could adopt the below
1. A build verification and acceptance tests for daily builds.
2. Practice code commit reports efficiently.
3. Run the automated regression tests.
4. Produce tests which cover component and feature level integration.

Have a short regression cycle atleast after two sprints which covers manual and automated testing.

Exploratory testing in Agile

Development cycles are short, Features often evolve or change quickly, Feature requirements are incomplete or bound to change. QAing the features in such short cycles needs effective planning.

Sanity testing or smoke testing would make things better in such situations. However, there are other aspects like feature integrations, performance, non-functional, test plan updates, etc are left at risk.

To a certain extent, automated regression tests would make a difference. Running these tests are every nightly build will boost up the confidence of the engineering. However, these tests would produce desired results when the features under test do not change drastically.

To keep up to the pace in such situations, Exploratory Testing would be considered as an important activity.

Exploratory testing focuses on new features which have been developed, Complex or new scenarios that have not been covered in manual or automated testing. It is a process oriented while adapting techniques and is limited to scope.

It is very similar to adhoc testing however, some key differences should be noted:

Adhoc Testing	Exploratory Testing
Doesn't have a any process/test case/Test scenario defined/preplanned to do it.  It involves simultaneous test design and test execution	Simultaneous learning, test design and test execution. It is a type of adhoc testing, but here the tester does not have much idea about the application, the tester explores the system in an attempt to learn the application and simultaneously test it
Advisable that adhoc testing is conducted by the same test engineer who has basically completed formal testing	Exploratory testing comes under experience based testing, the people having better idea about the application should be given with this responsibility

Exploratory testing produces a quick feedback on a functionality or the system. Helps to get familiar with the features, enhance the understanding of the system and hence testers efficiency is increased.

Good user experience

Two words are often used synonymously while describing a good user experience using an application.

The Usability and The User Experience(UX).

Both Usability and User experience are essential to the success of a application.

Usability focuses on goal achievement when using an application while a User experience focuses on design, human factors, accessibility, marketing as well as usability.

User experience(UX) focuses on having a deep understanding of users, what they need, what they value, their abilities, and also their limitations. Business goals and objectives of the application are also accounted.

Some of the common misperceptions
1. Thinking that because something is easy to use, it's a good user experience.
2. Thinking a beautiful design will magically make an application easy or intuitive to use.

When do we say that an application is usable?
An application which enables user to achieve their goals quickly, with minumum fuss or frustation and without error while been intuitive, simple or extremely learnable.

When do we say that the user experience is good using the application?
A usable application which emphasize on presentation, interactive behaviour, functionality, assistive capabilities and system performance.

Some factors to be considered to see if the application is usable and gives a good user experience
1. Adoptability
2. Learnability
3. Efficiency
4. Transparency
5. Desirability

These factors will be detailed soon...

Telerik Tests - Debug a coded step

There are challenges to debug Telerik tests that use both Recorded and Coded Steps.

Test Studio gives flexibility to debug Recorded Steps. However, Coded Steps cannot be debugged in Test Studio.

Coded Steps can be debugged using the Test Studio plugin in Visual Studio. Below are the steps to debug a coded step.


In the below test, there are 2 steps, wherein second step is a coded step that will be debugged.

Put a breakpoint in the line of code that you want to debug

Now, the important step is Open the Test Explorer which is located in TEST menu.
Navigate TEST -> Windows -> Test Explorer

In the Test Explorer, identify your test and Right Mouse Click on the Test and Choose "Debug Selected Tests" option.

This will hit the breakpoint that was placed in the coded step and you can debug your code.

Leverage Rollbase for Telerik Reports

In my previous blog titled Create custom plugins for Telerik Test Studio, i have mentioned how we can create plugins to customize Telerik Test Studio reporting.

The custom plugin is used to post the Telerik Test Studio reports into Rollbase wherein, we create Reports and Charts.

Screenshot of Rollbase Application:

Screenshots of the Charts created in Rollbase which gives a comprehensive and historical view of the reports:

 

Create custom plugins for Telerik Test Studio

If you are looking for a customized test run reporting or a custom bug reporting capabilities then, create a plugin (an extension).


There are few forums posts in Telerik forums which provides instructions to do. Here are couple of examples to create a custom plugin.

1. Customized Run reports

To customize run reports, the interface IExecutionExtension in ArtOfTest.WebAii.Design.Execution namespace should be implemented.

Following are the methods that needs to be implemented

a. OnAfterTestCompleted(ExecutionContext executionContext, TestResult result)

b. OnAfterTestListCompleted(RunResult result)

c. OnBeforeTestListStarted(TestList list)

d. OnBeforeTestStarted(ExecutionContext executionContext, ArtOfTest.WebAii.Design.ProjectModel.Test test)

e. OnInitializeDataSource(ExecutionContext executionContext)

f. OnStepFailure(ExecutionContext executionContext, ArtOfTest.WebAii.Design.AutomationStepResult stepResult)    

2. Customized Bug reporting

To submit a bug that the automated tests have encountered during execution, the interface IBugTracker in ArtOfTest.WebAii.Design.Extensibility.BugTracking namespace should be implemented.

Following are the methods that needs to be implemented

a. ApplyPersistableSettings(BugTrackerPersistableSettings settings)

b. ConnectionUI

c. ErrorMessage

d. GetPersistableSettings()

e. IsConfigured

f. Name  

d. ResetSettings()

e. SubmitBug(IBug bug)

f. OnStepFailure(ExecutionContext executionContext, ArtOfTest.WebAii.Design.AutomationStepResult stepResult)    

Once you have implemented, build the project to create a dll and the dll plugin has to be placed in the plugins folder of Telerik Test Studio installation. 

Automate your daily tasks

Things that you do daily like installing a latest build, creating a basic environment with which you can start working on your tasks. All such things should be considered for automating.

For me i found Apache ANT, Shell and Batch Scripting are good tools or technologies using which i could quickly automate.

These tools are such that, they don't need any extra efforts to get them in place. For example, if you are using batch scripting, you don't have to download any software to work on it. Similarly, Apache ANT is a tool which is readily available for most of the products that we work with.

Automating small and simple things benefit you speed-up your daily tasks, you learn more.

Re-use

In our daily work, we often hear or think about re-use. However, we don't spot opportunities for re-use and, the time and effort saving benefits that can be seen from re-use.

Re-use comes in many forms from a program created to facilitate the ease of future similar programs to automated test created in a way that can be easily reused time and time again.

Re-use can be a powerful and useful thing for everyone however, we should not get stuck trying to make everything re-usable, recognizing when and where to use it can come from trial and error, we don’t always get it right. If it’s not something you've thought of before then we should start looking for opportunities.

The trick for me at least is recognizing when the opportunity is there, with a little thought about future usage of something you can tell if it’s worth putting in a little extra effort upfront to take away the effort in future usage, sometimes it’s worth it other times its simply not.

Root Cause Analysis: Why and How

Do testers always have to provide root-cause analysis? No, they don’t need to and, in fact, all depends on a particular case. Every single testing activity is different from another and there are priorities you have to deal with as project resources are never limitless. There may be small features that don’t require that effort and dedication and there are large features that is quite vital to the product, root-cause analysis may be required.

However root-cause analysis is, indeed, a great thing, you should at least give it a try and here is why: it grants you with more information and data is pure power in a tester’s arms. After root-cause analysis testers do understand the feature under test in a significantly better manner. You can locate so many new marvelous things like critical areas you have never tested and you will gain additional scope during the analysis. And, surely, there will be more bugs for you to track.

Why isn't everybody doing it then?

This process is time consuming and not all projects are capable or willing to spend more additional resources on such tasks, especially if they are low-priority and the release cycles are very short. There are even times where some team members already know what hides in the root-cause making this activity useless.

How can I tell whether any root-cause analysis should be performed?

There are several factors, by measuring which you will be certain on whether the root-cause analysis activity is something you and your project can afford. 

First of all take a deep look at the defect. Is it worth is? 

Secondly go through your current test coverage as well as your test plan to make sure if you need this additional work in the first place. 

Then review defect priority to make sure if it requires additional attention and take a good look at your team/team members: are they up for more work helping you? Do they have time? Is it really necessary? 

And if all the mentioned above factors are satisfying you should go for root-cause analysis.

Finally, if you want to do it, convince with confidence!.

Source: http://blog.bughuntress.com/

Don’t Expect Magic From Test Automation

The primary reason for test automation is to free up QA time for interesting exploratory testing and to give confidence to the team that the application is still in good order as new changes are delivered.

Don’t expect automation to find lots of bugs. In fact, the number of bugs found by automation is always much less than manual and exploratory testing.

Few reasons Why Automated Tests Fail to Find Regression Bugs
It is widely believed that the purpose of automated tests is not to find new defects but rather find regression bugs as new features are developed. But even so, there are many occasions where regression bugs slip through that end up in production and really should have been caught by the automated regression tests.

Let’s examine the reasons why automated tests failed to find the regression bugs:

The Scenario Was Not Thought Of

Automated Tests are as good as the person who designed the tests. Automated Tests are a set of instructions that are executed against the target application. First, test analysts design test cases and come up with a list of scenarios. The scenarios are then scripted in a programming language to execute the scenarios automatically. Therefore, if a particular scenario was not thought of, it wouldn’t have been scripted to run as automated test.

The Scenario Was Thought Of But Was Not Scripted

It takes time to automate tests. Depending on the complexity of the tests, test engineers coding skills, flexibility of test automation tools and frameworks, some tests take a long time to automate and hence miss the chance to find regression bugs as new features are developed.

There Is A Bug In The Test Code Itself

There are situations where the automated tests do not really run against the tester’s intentions or assumptions. In other words the automation engineers made a mistake in coding the tests or did not insert verification points in correct places.

The Automated Tests Couldn't Execute Due To Environment Issues

This is particularly true when running System Tests via application UI, e.g. launching the application in browser. In such cases there are many dependencies on other applications, 3rd party or downstream systems, and if any of these are down or not responding or respond intermittently, the automated tests could not execute successfully and hence could not verify correctness of a particular test.

Poor Analysis Of Test Reports

After automated tests have executed, when there are failed tests, analysis is required to see the reason for the failed cases. This can take quite some time to analyze all the failed cases (many of which can be false positive). The analysis part is normally done manually and if the analysis is not done correctly, there could be genuine failures that are overlooked or masked by other issues.

Can you think of other issues why automated tests miss defects?

Source: http://www.testingexcellence.com/test-automation-tips-best-practices