The modern IT marketplace of the high technologies offers the multiplicity of great solutions for software testing.
In this article we review the pros and cons of the most famous web-based test management tools:
SpiraTest,
TestRail and
TestLodge.
Every of the mentioned services is the comprehensive web-based test management software that allows to control, monitor
and organize effectively the testing efforts of the QA department.
The main advantages of the aforementioned tools are:
SaaS delivery model
(allowing to avoid losses on installation and maintenance of the software),
simplicity of the user interface (allowing to manage all the test artifacts in one workplace),
a wide range of features (requirement management, test management, reporting management, etc.) and
ability to integrate with the large number of bug trackers and frameworks.
Outline
- licensing
- access control
- integration with external tools
- working interface
- requirement management
- test design
- test execution
- summary table
- conclusion
Licensing
All the considered applications are commercial and offer 30 days trial version for familiarization with the product.
Every of them presents its own licensing system, so SpiraTest and TestRail use the licensing arrangement
based on the number of users , while the licensing rules of TestLodge are based on the quantity of activities
(the amount of test plans, test cases and test runs) and do not depend on the number of users.
Access control
TestRail provides fine grained access control which allows to edit pre-installed roles, add custom roles
and combine users into groups. Each role allows to set permissions for configurations, milestones,
test-cases, attachments, test runs, test-results and reports. SpiraTest offers very similar access control
system which allows to edit preset roles and create custom roles with a variety of permissions.
As for TestLodge, the access control of this app allows to assign admin role to a user and create custom roles
which define read/write permissions for test plans, requirements, test suites and test runs.
Integration with existing tools
All the mentioned test management applications support integration with a wide range of bug tracking systems,
frameworks and add-ons. But, only SpiraTest offers its own built-in issue tracker, TestRail and TestLodge refused this
idea, since they offer the large number of external bug trackers. The most popular of them are listed below:
| Integration with bug trackers | |||
| SpiraTest | TestRail | TestLodge | |
| Built-in issue tracker | + | not implemented | not implemented |
| Assembla | not supported | + | + |
| Basecamp | not supported | + | + |
| Bitbucket | not supported | + | not supported |
| Bugzilla | + | + | + |
| Fixx | not supported | + | not supported |
| FogBugs | + | + | + |
| Gemini | not supported | + | not supported |
| GitHub | not supported | + | + |
| IBM Rational ClearQuest | + | not supported | not supported |
| Jira | + | + | + |
| LightHouse | not supported | + | + |
| Mantis | + | + | + |
| OnTime | + | + | not supported |
| Pivotal tracker | not supported | + | + |
| Rally | not supported | + | not supported |
| Redbooth | not supported | not supported | + |
| Redmine | + | + | + |
| RTC | + | not supported | not supported |
| TFS | + | + | not supported |
| TRAC | not supported | + | not supported |
| Trello | not supported | + | + |
| Unfuddle | not supported | + | + |
| Vault | not supported | + | not supported |
| Youtrack | not supported | + | not supported |
Working interface
TestLodge and TestRail provide the well-organized and simple UI, allowing to capture quickly
an idea of a project. However, this simplicity may be criticized by experienced users who want
all the sheaths at once. As for SpiraTest, this tool offers intuitive UI which includes a lot of filters
to narrow down the number of shells on the workspace, but at the same time this app suffers
from the multiplicity of the “edit” and “update” buttons, that force user to make a lot of redundant “clicks”
to finish any edit and update operation.
Requirement management
TestLodge and SpiraTest offer built-in requirement management (RM) modules which allow to perform CRUD operations on
requirements, attach files and assign test cases to requirements.
TestRail does not provide built-in RM service, but supports integration with external services, like Jira.
As for the ability to generate traceability matrix (a document that shows the test
coverage of requirements), all the reviewed tools support it quite well.
Test design
All the reviewed apps allow to perform CRUD operations on the test-cases, but TestLodge and TestRail
implement simple and fast method of test case management, which allows to add a test-case with the minimum number of details
(title, description, steps, expected result).
The next point to review is a search functionality, here SpiraTest and TestRail are taking the lead, since TestLodge
does not support this feature at all. The difference between SpiraTest and TestRail is in the fact that SpiraTest offers
the global search through all the projects and test artifacts, and TestRail offers
search only among the specific test artifacts and does not support searching by keywords.
As regards import/export, TestRail and TestLodge provide easy in use import/export without configuring,
while SpiraTest requires to install and configure specific add-on.
| Import/export formats of the test-artifacts | |||
| SpiraTest | TestRail | TestLodge | |
| Import | MS Excel; MS Word; MS Project | XML | Google spreadsheet |
| Export | MS Excel; MS Word; MS Project | XML; CSV; Excel | XML; HTML |
And lastly, the document attachments. All the reviewed applications allow to attach any kind of files to test cases,
but in contrast to SpiraTest and TestRail, the number of attached documents in TestLodge is limited in
accordance with the selected plan (non-trial).
Test execution
SpiraTest and TestRail display the execution results as a tree, while TestLodge presents test results in the row mode
(test runs are listed in the rows which are located one by one on the main page of the service).
Both types of the interfaces are intuitive and easy in use, but tree-based structure is more comprehensive,
since the critical and valuable data are displayed clearer.
As for the test results tracking (document that displays status of test cases for a project
and allows to analyze and monitor how the development work is going), here SpiraTest is a bit behind, because TestRail
and TestLodge have deduced this feature into a separate unit, allowing to trace the history of launches and tests
editing more precisely and clearly.
Summary table
| SpiraTest | TestRail | TestLodge | |
| General points | |||
| Trial mode | 30 days | 30 days | 30 days |
| Licensing | based on the number of users | based on the number of users | based on the number of activities |
| Requirement management | |||
| CRUD operations | + | only through external RM tool | + |
| Complexity of adding a requirement | too many fields | only through external RM tool | simple |
| Structure of requirements | tree-based structure | depends on external RM tool | row-based structure |
| Traceability matrix | + | + | + |
| Integration with external RM tools | + | + | + |
| Test design | |||
| CRUD operations | + | + | + |
| Complexity of creating a single test case | too many steps | simple method | simple method |
| Search functionality | global search | limited search | not implemented |
| Custom fields | + | + | + |
| Import/export | + | + | + |
| Documents attachment to the test case | + | + | limits are set in compliance to the selected plan |
| Test execution | |||
| The execution interface | tree-based structure | tree-based structure | row-based structure |
| Manual execution | in a browser window | in a pop-up window | in a browser window |
| Test results tracking | + | + | + |
| Defect management | |||
| Inbuilt (own) defect management system | + | not implemented | not implemented |
| Integration with issue trackers | + | + | + |
Conclusion
All the considered tools are well-organized test management services that offer a wide range of the
capabilities and features, allowing to cover the most part of users’ needs.
TestLodge and TestRail offer thought-out and convenient UI which in combination with a sufficient
number of the features, allows to resolve the majority tasks of small and medium-sized projects.
As for SpiraTest, it is powerful but a bit complex test management software,
which can be suitable for big and complicated projects.
Hopefully, the provided information will help you to select the most suitable test-management tool for your needs.