Like many software testing service companies, we often subscribe to several of the leading cloud based infrastructure test automation services to meet our clients specific multiplatform/multi device regression testing requirements. These solutions provide us with the ability to perform software testing using various scenarios and platforms while eliminating the investment in buying and maintaining actual equipment.
The associated cost for these services vary – but an example of a recent quote from one of the leading platforms was, “$200 per month for 2 parallel executions for approximately 33 hours of automation.”
In addition to comprehensive feature/device testing, our clients require solutions that reduce the test execution time as much as possible. Having delayed execution is the one specific challenge we have encountered with these cloud services, because although they provide a rich number of features and flexibility – the execution time often increases from 2x to 4x of the local execution time (for the same script). This standard seems to be acceptable from one of the leading providers, with this response provided to our inquiry:
“Yes, that is pretty ideal. You should expect your tests to take ~2x – 4x longer to run in the cloud. In the case you described below they are running better than 2x slower, which is great. If your tests start taking more than 4x local execution let us know and we will investigate.” –support group answer to execution time inquiry.
Of course, there are understandable reasons for this longer processing time – but the time extension is unacceptable for several of our projects – which require greater speed for execution of smoke test.
In order to address this issue for a client, we recently implemented our own grid configuration in the cloud. We acquired a machine in AWS with Windows with a configuration of type t2.medium (monthly price is around $50 depending on your configuration), and set up the entire grid. You can see all the required steps to configure it on the following link.
This specific project currently has 13 test cases to be executed in the smoke test which is why we decided to use the same machine with the concurrent roles of hub and slaves. Due to the feasibility to maintain scalability in AWS, we anticipate it will be easy to configure additional nodes to implement additional test cases in the future.
Of course, we are required to maintain the execution environment, keeping the grid up and running, but to date, this is simple to manage by using a batch script that will create the hub and nodes consoles. Another required task which is normally provided by the service is the implementation of “taking screenshots” feature – which you can achieve using the following code:
I also recommend the following library org.monte.screenrecorder.ScreenRecorder and follow this link.
This set up took no more than a day to complete. The test case executions in our cloud grid realized on average 10% to 15% of the additional time of execution compared to local execution time. The total execution time of the smoke test was reduced as well because we now have up to 5 instances running in parallel.
Although we are responsible for maintenance, with this one-day implementation of our own test grid, we are:
- saving money,
- increasing the number of parallel tests,
- reducing the execution time for individual and smoke test,
- increasing the availability of automation minutes (unlimited now)
Alex Chaves
GlobalNow IT QA Manager
SIgn up