Should I Leave Xray? An Honest CEO Interview on Migrating to ATM

We asked our own CEO the questions a skeptical Xray user would ask: pricing, security, automation gaps, and honest answers on who should stay put.

Before we start

I work at Appsvio, the team behind ATM. Full disclosure: this isn’t a neutral interview. Instead of asking “what makes our tool great,” I played devil’s advocate for Xray. It’s a tool we respect, and it still holds its own in several areas. I asked the questions you’d need to answer before proposing a migration to your QA team and the person approving the budget. If you finish this and still decide Xray is the best fit, fair enough. The goal is an informed decision, not a sales pitch.

I sat down with Krzysztof Skoropada, CEO of Appsvio, to get those answers.

Who Should (and Shouldn’t) Migrate From Xray to ATM

Which organizations shouldn’t migrate from Xray to ATM today, and when does it actually make sense?

The real dividing line comes down to whether an organization needs to run entirely on cloud, Forge-native solutions. Beyond features and price, the biggest factor is compliance: where your data is processed and stored. For European teams, this is often non-negotiable, dictated by regulators or internal security policies. On the other hand, US companies would like to store data in North America.

Xray’s cloud data lives in just three regions: the EU, Australia, or the US. It has no dedicated UK or Switzerland data center. ATM supports every region Atlassian offers, including the UK, Frankfurt, the rest of Europe, Singapore, Canada, Japan, India, South Korea, and Switzerland. If you have strict residency requirements, that alone makes a migration worth considering.

Then there is the Data Center question. Atlassian has announced its end-of-life. Some customers have an extension until 2032, but after that, it is gone. If you are on Xray Data Center and planning a move to the cloud, whether this year or in a few years, it pays to evaluate the whole market rather than automatically re-buying what you already have.

If data residency and vendor risk matter to you, or if you are dealing with functional gaps and an outdated UI, it is time to look at alternatives.

Is Xray Still Worth It? Installs, Maturity, and the Data Center End-of-Life Deadline

Say I’ve been an Xray user for years and we have our whole test history in it. Why would we throw that away for an app with only 113 installs, while Xray has 25,000+?

Installation numbers mostly reflect time on the market. Xray was an early mover, so it had years to build its base. It is also highly configurable and supports automated testing well, points we will get into later. If what you have works, staying put might be enough. You can keep the status quo and accept its limits.

The issue arises when teams notice better options; tools with smoother workflows that feel deeply integrated into Jira. That is when you start weighing the migration effort against the daily benefits.

Because Xray is so large and mature, its development cycle has naturally slowed down. Recent Marketplace reviews often cite slow feature updates and inconsistent support. Our roadmap, on the other hand, is driven directly by customer feedback. We also guide clients through the entire process, from initial evaluation to migration and ongoing maintenance.

Switching tools always carries some risk. A team looking at newer options like ATM needs to measure the short-term migration cost against long-term gains. In the end, it comes down to whether you trust a newer vendor to deliver a better daily experience.

Xray runs on Jira Cloud, Server, and Data Center. ATM only runs on Cloud because Forge requires it. What does that mean for an organization considering migration?

With Data Center reaching its end-of-life, moving to the Cloud is inevitable. At this point, there is no strategic reason to build new products for the Data Center.

For us, that has been an advantage. We never had to split engineering resources across multiple architectures. ATM was built exclusively for the Cloud from day one, allowing us to focus entirely on a single platform.

The same goes for Forge. While many newer apps build on Forge, most still store their data on external servers. ATM was designed Forge-first, making it one of the few test management tools that fully uses Atlassian’s native security and storage infrastructure.

View on sample test execution in ATM.

Runs on Atlassian, Data Egress, and the Xray “Issue Bloat” Myth

You mentioned “Runs on Atlassian.” ATM have that badge, while Xray doesn’t. What does that actually change for the client’s security team?

For a security team, Runs on Atlassian is a key Marketplace distinction because it explicitly addresses data egress. Egress means the app can send data to external services outside Jira, such as third-party analytics or session recorders, where you may not want your data going.

Having this badge means there is no data leaving Atlassian’s infrastructure. To my knowledge, no other test management app on Jira holds it today. For a security team that understands Forge, approving an app with this badge is straightforward. There is no vendor-hosted infrastructure to audit. From a compliance standpoint, it is easily the most valuable badge on the Marketplace.

Xray stores tests as Jira issues, and at scale, that creates a known problem: “issue bloat.” Is that a real pain point for the clients you talk to, or has it become more of a myth?

A few years ago, it might have been a valid concern. But consider the numbers people usually cite when discussing issue bloat, often around two million test cases in a single instance.

In practice, almost no team manages two million actual test cases. Two million test executions? Yes, that is a realistic figure for an active organization. But two million individual test case issues clogging a backlog is not a realistic setup. The extreme scenario people worry about rarely happens in the real world.

Atlassian has also made major progress on Cloud performance and scalability, so that kind of volume is no longer the issue it used to be. On our end, we have clients running close to 100,000 test cases in ATM without performance issues, organized neatly across folders. Scale simply is not a bottleneck for us today.

Xray vs ATM: Feature Comparison

Xray positions itself as an automation and BDD powerhouse. How does ATM compare, and what happens to our CI/CD pipelines if we migrate?

Credit where it is due: Xray’s automation support is strong. They handle Gherkin and import results from JUnit and other frameworks out of the box. We are currently building Gherkin support, but it is not available yet. This is an area where we are actively catching up.

For teams with heavy automation needs, we share our detailed roadmap so they can decide whether to wait for upcoming integrations. But if full automation support is a non-negotiable requirement from day one, ATM is honestly not the right fit today.

Test step versioning is sold separately in Xray Enterprise, but included as standard in ATM. What does that mean for the actual price difference?

Test step versioning requires Xray Enterprise, meaning you have to buy a higher-tier license just to get that feature. At the 100-user tier, the combined cost (for both the Advanced and Enterprise editions) comes out to roughly $22 per user per month. ATM, at the same scale, costs around $2 per user.

That said, Xray Enterprise includes other advanced features beyond step versioning. It pays to look closely at what each plan includes and decide if you actually need those additions. If you do not, the cost difference becomes a major factor in the evaluation.

Xray has separate REST and GraphQL APIs, a ready-made Reporting Center, and a Document Generator. ATM’s API runs on Forge’s apiRoute: safer architecturally, but doesn’t that just mean narrower? Do you have real functional parity?

Setting automated testing endpoints aside, we are at roughly 80% endpoint parity today, and we continuously expand that list based on direct customer feedback. Building our REST API on Forge’s apiRoute module ensures zero data egress. When a client calls any endpoint, that data traffic stays entirely inside Atlassian’s infrastructure from start to finish. It is the strongest approach for organizations prioritizing data security and Atlassian-managed infrastructure.

Xray lets you customize fields, statuses, and entire workflows, often representing years of fine-tuning. How much flexibility does ATM give you versus a more rigid, pre-built template? Can I add custom statuses and fields, or am I locked in?

Both tools offer the same level of flexibility when it comes to native Jira workflows and custom fields. In both Xray and ATM, elements like Test Cases, Test Plans, and Executions are standard Jira issues. That means you can build custom approval flows with statuses like New, Approved, or Archived, and apply third-party custom fields across all objects.

Where ATM takes the lead is in how naturally those workflows operate. Being Forge-native and built specifically for the Cloud allows us to support shortcuts Xray cannot match. For example, when creating an Execution directly from the standard Jira issue creation screen, you can select Test Cases and Test Plans right away. Xray simply cannot do that.

Is Appsvio Test Management (ATM) SOC 2 Type 2 Certified?

SOC 2 Type II is an ongoing audit, not a one-off check. Which period does your report cover, and have you had any security incidents since launch that a prospect should know about?

Appsvio has been SOC 2 certified since 2023, and our report is renewed annually. The audit itself evaluates our procedures and controls over a designated three-month window each year to ensure full compliance with SOC standards. We have had zero security incidents. Furthermore, because our entire data management architecture is built directly on Atlassian’s infrastructure, the risk of future incidents is inherently minimized. It gives both us and our clients a high level of security.

Sample ATM repository.

Xray Pricing vs ATM: Real Per-User Licensing Costs Compared

Xray licenses are based on your entire Jira user count, not just your QA team. How does the pricing compare for a company with 500 Jira users but only 10 testers?

Both tools follow the standard Marketplace model set by Atlassian, where licensing scales with your total Jira seat count. However, Xray introduced a significant price hike recently, making it extremely expensive alongside competitors like Zephyr.

At 500 users, Xray Advanced costs over $21,000 annually. ATM, at that same 500-user tier, comes in under $5,000. Even with a small QA team, you are paying roughly a quarter of the price for the software license alone.

Does that license gap make pricing your strongest argument for migration?

It is a major factor, but the software license is only part of the equation.

First, you have ongoing maintenance. With legacy tools, consulting fees and partner support often end up exceeding the license cost itself. Moving to a modern, cloud-native tool allows internal teams to manage their processes independently.

Second, there is the upfront migration cost. Moving 100,000 test cases from Data Center to Cloud requires planning and execution, which takes time and budget.

When you look at the total cost over a two- to three-year horizon, factoring in licensing, migration, and maintenance, moving to ATM easily pays for itself.

How to Migrate From Xray to ATM: Free Migration, Timeline, What Transfers

Free migration from Xray to ATM: what exactly gets moved, and what stays behind? Do execution history, attachments, and requirements links transfer over?

We focus primarily on migrating Test Cases through a straightforward process. If your Test Cases include attachments, we first move those issues using Jira’s native import and export tools to keep the files intact. Once they are in the Cloud, ATM’s importer processes the test steps.

If there are no attachments, you can just use ATM’s importer directly. The one thing to take care of is exporting your Test Cases in the right format, and that’s where we can help, formatting an export from, say, Zephyr or Xray, so the import goes smoothly.

How long does a migration of 5,000 test cases actually take?

Because ATM is a Forge app running entirely within Atlassian’s infrastructure, import speeds depend directly on Atlassian’s REST API rate limits. For a batch of 5,000 test cases, the process takes approximately 60 minutes.

Switching tools affects more than just QA: developers, PMs, and DevOps also touch tests in Jira. How many teams beyond QA actually need to adapt, and who accounts for that in the project budget?

Putting a specific price tag on team adaptation or process changes is genuinely difficult. What I can clarify is which roles actually interact with a tool like ATM. Beyond QA, business analysts use it heavily. They own, write, and update requirements, then link them directly to Test Cases to maintain coverage.

The other group is business leadership. To support them, we are building a feature, not yet released, that ties multiple Test Plans together under a single layer, which we call a “Test Campaign.” It acts as a master test plan, allowing executives to track progress across large, multi-application releases and forecast launch timelines from a single view. That is a major touchpoint between business stakeholders and test management, and Xray has no equivalent for it today. It is an area where we are clearly ahead.

Xray vs ATM: Where Each Tool Still Wins

What does Xray do better than ATM today, and when do you expect to catch up?

Automated testing support, without question. Xray is well ahead of us there. We do not have those capabilities yet, to put it plainly. We expect to launch a comparable set of integrations next year.

Where is ATM’s biggest advantage today, and what will success look like a year from now?

Our main strengths are data security, intuitive workflows, and a modern user experience.

As for the future, I would push back on the idea of “beating” Xray. We are not chasing a larger install base or sheer customer count. Success for us is much simpler: making sure that every company using ATM sees real improvements in their daily testing processes. Customer satisfaction, not market size, is our true metric.