Istanbul automated testing

One of the most exciting feature of the ServiceNow Istanbul release, is the new Automated Testing Framework (ATF).

Why so excited over automated testing?

• Custom test frameworks are very expensive to implement.
• Most customers only do manual tests.
• Every time customers upgrade their instance, weeks of works are needed to test against regressions. The time spent bug-fixing is often tiny compared to the large amount of time spent on manual testing. Anything that has automated tests will consume almost no resource at all and will benefit of better reliability.
• Because of this resource burden many customers don’t upgrade their instance as often as they could. Instead of having a steady process of continuous improvements, they focus all their resources over testing and ignore new functionalities that could benefit their processes and users.
• ITIL best practices recommend continuous improvements. Yet, customers don’t like to implement changes, or experiments with new features, because it is too heavy on costly resources when not automated.
• Major UI changes tend to break documentation made for testers, or break tests implemented in custom test frameworks, like Selenium. Istanbul’s automated tests are UI agnostic.
• We now have a free, out-of-the-box, automated test framework specialized for ServiceNow.

Having spent two years as a DevOps specialist, I’m betting this will be a great time and cost saver for customers. DevOps logic embrace continuous improvement, as with Agile Software Development. But to allow the desired flexibility, you need to have automated builds, tests, and delivering. In software factories, large amounts of developers are constantly improving the software, with automated test running multiple times each day. Any detected regressions are fixed immediately. Every time a build is stable and error free, you can push it to production as a new minor release. Major releases use the same logic, but come with improved documentation or training for end-users. This is exactly what ServiceNow does for its own platform.

Now let’s dig over the specifics of Istanbul’s implementation. Keep in mind this is a 1.0 version. ServiceNow had to decide where to concentrate their first effort; they focused where the return on investment was the biggest: Forms and server based test. Additional features will have to wait for future releases.

What’s included?

• Client side UI testing o Same browser requirement as UI15
  • Take a screenshot in case of failure
  • Wait for asynchronous effects
  • Capture client-side logs, alerts, etc
  • Support the following automated actions:
    • Open a new form
    • Open an existing record
    • Set field value (quite similar to Templates UI)
    • Click modal button
    • Field values validation
    • Field state validation
    • UI Action visibility
    • Submit a form
    • Click a UI Action

• Server side testing
  • Extensible by customers
  • Runs entirely on the server
  • Can be mixed and matched with UI steps
  • Include powerful Jasmine JavaScript framework for behavior-driven scripted testing
  • Support the following actions:
    • Impersonate
    • Record Insert / Update / Delete
    • Record Query
    • Record Validation
    • Replay Requested Item (Required, because Service Catalog UI testing is not supported)
    • Run a server side script
    • Log

The tests themselves are tracked in update sets. They are customer configurable, obviously. They can be grouped hierarchically, in test suites. Some elements from previous test can be defined in subsequent tests.

A typical test suite would be something like: Impersonate user XXX, create a ticket, impersonate ITIL user, open previous ticket, check that automatic Assignment group is correct, check that caller is XXX, click on Resolve button, ensure that Close Note field is mandatory…

At the end of the execution, any modifications in the instance are reverted: Records that have been created are deleted, updates are cancelled. You can execute your test as many times as required without polluting your instance.

Current limitations

• ATF isn’t designed to serve as performance or load tester.
• In the current version, I did not find any out-of-the box settings to schedule periodic automatic tests.
• UI testing of Service Catalogs, Service Portals, CMS, Record producers, login pages is not supported.
• Not available for Express instances

Who should design tests

Two roles have been added to Istanbul:

• atf_test_designer: Can design and publish test.
  Could be developers or technical business analysts.
• atf_test_admin: Can create test items to for test designers.
  This is for hardcore developers specialized in automated testing.

Both roles grant impersonator role so you should grant these only to users who are cleared, if there are confidential information on your instance.

Suggested next steps

You’ll need to enable the Automated Test Framework plugin, if your instance was created before the Istanbul release.

You should ensure that every new implementation on your instance includes as many functional tests as possible, implemented directly via ATF in your development instance by the software developers. They should be included in your release’s update sets.

I also suggest that during your next major upgrades, you spend resources to automate as many of your manual tests as possible. The return on investment should be obvious on you’re the following upgrades.

Are you planning to use Automated Tests? Did you already start to use it? Do you already use another framework? In that case, here could be a nice place to exchange on our experiences.