Oct 16, 2018
Our Opinion that leads us to use Xray
There is a lot of test case management out there, and some of them even well known for users to run tests especially to stored on JIRA. However, in our workplace, there’s a newcomer that seems to captivate the QA’s heart and now being used. Of course for some, we wonder would one be better than the other, why the changes or are they similar to the one they used in their old workplace. Out of 10 (ten) management tools, 3(three) come close. Being new yet able to captivate the heart of QA’s in our place, how does Xray stand out?
Documentation
Zephyr and Xray can do traceability reports and comes with its own project and dashboard reports. Xray, offer to show the Latest Test Execution and Requirement Traceability which made it easier for new member and non-QA team.
The system of Zephyr is clear, with all test tools, like a dashboard, test case, test execution report, release management, and link to Jira for an issue. However, Zephyr does not allow to import the test results and the Test Case format cannot be customized. Just two types of traceability reports are available:
1.Requirements → Tests → Executions → Defects
2.Defects → Executions → Tests → Requirements
Test case importing from an existing excel sheet is possible but not easy. Grouping large amounts of tests in Zephyr within JIRA is challenging and filtering and searching are quite hectic when you have lots of test cases. There is a suggestion to overcome this such as using eazyBI (add-on). In Zephyr, you can still search the tests themselves using JQL as a test is just another issue type.
The downside, however, Zephyr is limited that it cannot show you the latest execution of any given test regardless of which test cycle it belongs to, you can only see the latest execution for any given Cycle. This may cause users to clone tests which mean you end up with lots of duplicate tests and nobody knows which one is the latest.
Testrail reporting and getting set up initially not super intuitive — of course, you could found the answers online but the need to search for how to do setup of test runs and making reports initially could be better explained in-app or more intuitive, this will make a new member of the team find it challenging for the first time, especially since it does not have the inability to give other than QA team the view only access in tools such as Jira. It requires giving them a separate login just to see associate test cases to a Jira card.
Also, pulling a straightforward report (how many test cases passed vs. failed) may be a bit cumbersome.
Testrail could really use far better search capabilities. Once your test suites and cases get large enough it becomes really important to be able to search, currently, the search capabilities are not that nice enough to create that potential. The API integration is not extremely extendable or scale-able, the reporting mechanism is weak, groupings and numbering of test cases etc. get very confusing for the new team member.
Digging further into details is where they differ. Xray uses 4 (four) issue types — Test, Test Set, Test Execution and Pre-condition. As for the search capabilities, Xray uses native JQL so you can search the Tests or Test Sets within a Test Execution — this will provide you get meaningful reporting from groups of tests. Between those three, it is the easiest tools to use for reporting and searching. Since it also could create a pdf simple enough for non-QA team to see the report.
Automated Testing
As enterprises become bigger with tons of features, automated testing becomes essential. If you need to execute a hundred even thousands test for your weekly releases, an automation script integration could help a lot, and Xray has a native cucumber automation build in. Which means testing can take hours instead of days (nor weeks) with no need for another add-on. Xray also has Pre-conditions worth noting since it allows for specific criteria to be added into testing(e.g: a user has to be the member that has X tier).
Zephyr does not do automated testing as an add-in like Xray and you might need a way around such as “ZBot” for Zephyr (this is an additional add-on which you must purchase.) and this might be challenging for a new member who is not used to the tools. On the other hand, TestRail does support automation process on its own dashboard but it does not support automation script integration with Jira.
PRICING:
Assuming a smaller or new start-up organizations there are little differences between Xray and Zephyr for the term of price since both are cost $10 up to 10 users. TestRail, on the other hand, cost more. 30$/month/user so, in term of price especially for a larger organization who has more than 10 users, the advantages become apparent for Xray.
source TestRail: https://www.gurock.com/testrail/pricing
source Xray: https://marketplace.atlassian.com/apps/1211769/xray-test-management-for-jira?hosting=server&tab=pricing
source Zephyr: https://marketplace.atlassian.com/apps/1014681/zephyr-for-jira-test-management?hosting=cloud&tab=pricing
Basically, Xray is cheaper than the other two when applied to medium (or large) size organizations.
Conclusion:
Well on our opinion this is why we end up using Xray tools. I know it’s not that specific and might not help at all, but hopefully it might shed some light.
This table below are also the source for creating this article (sorry for the Indonesian word’s at the end of the table)
Warm Regards
Emile Francois
