PCF Controls — Testing with Playwright

13 min readApr 24, 2024

In my previous article on this subject, I demonstrated how we could set up a PCF control for use in Microsoft Dynamics 365, and then how to implement some basic unit tests.

In this article I want to show how we can set up integration tests to validate the behaviour of our control. Integration tests differ from unit tests in a number of ways:

Unit tests should always be self-contained with no external dependencies. (They should be runnable as part of an automated build process)
Integration tests may rely on external resources (in this case a Dynamics CE test instance containing a form that makes use of the control we are trying to test)
The integration test needs to try to ensure that any external dependencies are in an appropriate state for the test.

Also, apologies if this article runs on for rather longer than I would like — CRM, UI testing and dates and timezones has turned out to be rather more “interesting” than first expected, but some good lessons to be learnt here.

Playwright PCF Project

The code used in this project is kept here in github: https://github.com/rogerhillgsy/SpecificDateTimeControl

Playwright Setup

The main purpose of the Playwright platform is to support end to end testing of browser based applications. It follows the approach taken by Selenium and EasyRepro and offers code based setup of tests, but includes tooling to record user UI actions (as code) and subsequently to inspect and debug test execution and test failure logs, images and videos.

Test can be run interactively, or as part of a CI/CD pipeline.

Playwright setup is described on the website https://playwright.dev/, but the basic steps are: -

Install the Playwright for VS Code extension.
Run “Install Playwright” from the VS Code command palette.npx
From the VS Code test pane run the sample test “example.spec.ts” (This runs a very simple playwright test that accesses the playwright.com website.)

Authentication for Dynamics 365/ Power Platform

We have now established can run a test against the https://playwright.dev/ public website, which does not require any kind of authentication.

How do we test against a Dynamics CRM model driven app that will require authentication steps such as login, password and MFA?

There are currently two main approaches to this:

Add a “software mfa device” to your testing environment. This is described here and allows your tests to complete the MFA login and authentication process.
Store the browser authentication state locally and reuse it each time your test runs. This completely bypasses the authentication screens.

Each approach has its advantages. The virtual mfa device approach allows the authentication mechanism to be tested, but the password and OTP code for the account will still need to be stored securely.
On the downside, the authentication process does increase test setup time, and introduces an additional element of “flakiness” so I have chosen not to use this method here.

The second approach of storing the browser authentication state is easier to use when running interactively and has the advantage that it completely bypasses the login/authentication process which can speed up tests quite significantly (and also avoid any “flakiness” in the authentication process).
The browser state file can be stored securely (in a keyvault for example) and used in a CI pipeline.

We will use the second approach in this article, it is described in more detail here: https://playwright.dev/docs/auth.

An important component of this approach is the dotenv package (https://github.com/motdotla/dotenv). This stores environment values and secrets (such as a password) in a local .env file (which is excluded from git checkins through the .gitignore settings)

The login.setup.ts test script uses the contents of the .state file (if available) to configure the browser (without needing any additional authentication steps). If the .state file is not found it will take the username and password from the .env file and use an interactive browser session to login.
Once logged in the script stores the browser state in the .state file (which is also excluded from git commits).
The next time a test runs, rather than trying to log in again, it will use the stored browser state from the .state file.

Running the playwright tests

Following checkout of the SpecificDateTimeControl project, the following steps should run the playwright tests:

SpecificDateTimeControl> cd .\playwright-tests\ 
SpecificDateTimeControl\playwright-tests> npm update
SpecificDateTimeControl\playwright-tests> copy .\.env.template .env

The .env file needs to be edited and configured for your environment (with username, password and environment url).
Then run the tests:

SpecificDateTimeControl\playwright-tests> npx playwright test   --project chromium
Running 32 tests using 6 workers
[chromium] › integration\api-test.test.ts:7:5 › Get WhoAmI response from server
...
  1 flaky
    [chromium] › fixture\interaction-test.test.ts:42:13 › Pacific/Auckland timezone - Test data transfer between OOB date control and specific control and vice versa. › Specific date control to OOB
  31 passed (3.6m)

Serving HTML report at http://localhost:9323. Press Ctrl+C to quit.

Note that depending on your environment you may need to carry out multi factor authentication when you first run the tests.

The First Test

The first test we will look at accesses the WhoAmI endpoint on the CRM API. This avoids the CRM UI completely — and serves to illustrate how we use the stored state from the browser.

import {test, request} from '@playwright/test';


test( 'Get WhoAmI response from server', async({page, baseURL}) =>{
    const context = await request.newContext({
        baseURL: `${process.env.environmentUrl}/api/data/v9.2/`,
        storageState : ".state"});

    const response = await context.get("WhoAmI");
    var body = await response.body();

    console.log(body.toString())
});

When run, this outputs the JSON result of the WhoAmI call.

{
  "@odata.context": "https://xxxxxx.crm11.dynamics.com/api/data/v9.2/$metadata#Microsoft.Dynamics.CRM.WhoAmIResponse",
  "BusinessUnitId": "d103ac4b-c6e7-ee11-a203-0022481b53c4",
  "UserId": "ef486882-91f2-ee11-a1fd-0022481a8b6e",
  "OrganizationId": "00000000-32e8-ee11-a1f8-7c1e521d41aa"
}

This API based approach can be useful if specific state in the CRM system needs to be set up or validated before a test is run. (And it avoids any flakiness that might be introduced by the CRM UI)

The playwright.config.ts file defines the “setup” project that deals with authentication (if required). All of the other projects (chromium, firefox, etc) have a dependency on this setup project. This forces the setup/authentication process to be checked before our own test runs.

Test our connection to CRM

Next, we’ll start with a very trivial tests that connects the browser to CRM and checks the title of the resulting page.
Remember that this runs in the context of our test user’s account.

test("Open Interaction App", async ({ page }) => {
    await page.goto(process.env.environmentUrl!!);
    await expect(page).toHaveTitle(/Accounts My Active Accounts -( Power Apps)?/);
});

PS C:\Work\Medium\SpecificDateTimeControl\playwright-tests> npx playwright test interactions.test.ts -g "Open\s+Interaction\s+App$"                   

Running 4 tests using 3 workers
  4 passed (19.7s)

To open last HTML report run:

  npx playwright show-report

PS C:\Work\Medium\SpecificDateTimeControl\playwright-tests>

This is a very simple test that displays the initial page when the test user accesses our CRM system. There are a few points to note here:

If the test user changes their default screen this will break the test(!)
The title of the page seems to vary depending on the browser used and race conditions within the CRM UI. (i.e it may or may not end with the words “Power Apps”.)
The test is a single test that causes 4 separate tests to run across 3 workers. What does this mean?
We have three workers because our (initial) playwright.config.ts file defines projects for chromium, firefox and webkit. (1 worker for each browser).
We have 4 tests, because each of the other tests depends on “login.setup.ts” which is run once to ensure the browser .state file exists. Following this our single test is run three times (once each by three different browsers)
This means that we have managed to run our UI test across 3 separate browser platforms with minimal additional effort.

As it happens the webkit (Apple Safari) worker turn out to have several issues — Safari on Windows is no longer supported, and the actual behaviour of date fields in this windows version of Safari diverges considerably from the behaviour in modern version of Safari on MacOS and iOS. For this reason, I have commented out the webkit worker (in playwright.config.ts) for most of my tests.

Test the PCF control

To test our PCF control, we will create a CRM/Dataverse model driven form that exposes the control.

The same date field is displayed twice on the form, once using the out of the box control. The second time using our “Specific Interaction Date” PCF control.

A date and time entered into the OOB control will be reflected in our PCF control.
Vice-versa — a date and time entered into the PCF control will be mirrored in the OOB control (but only once a valid date and time is entered into our new PCF control).

We can now use Playwright to record our interaction with the form. We can then set up a test and run it from the terminal in VS Code. This test will connect to CRM, create a new interaction record and then fill in various values to see if the new PCF control works as expected.

This initial attempt produced the tests in test/integration/interaction.test.ts. While these tests work, they are hard to follow and are cluttered with a lot of boilerplate.

Note that these tests can all be easily run from the VS Code test explorer, or from the command line. It is also possible to select which browser to run the tests on, and the tests can be run in headless or headed mode.

Flakiness

One thing to be aware of is how flaky UI tests can be. Consider this view of the test explorer: -

Notice that the unit tests have run successfully, but the playwright tests seem to be failing in places. These failures tend to be rather random, and in many cases are simply due to the CRM system taking too long to respond, race conditions between the tests and the CRM system, or flaws and race conditions in the underlying model driven framework which are exposed by the challenging conditions we are testing under here.

What is worth noting is that if these tests are retried individually they will almost always pass (eventually)

The problems that affect interactive tests also affect tests run in a CI pipeline, so a degree of discretion needs to be used in interpreting the results of these tests. Usually this means configuring playwright to allow a test to be retried two or three times before it is marked as failed.

Ideally, we would write tests that were not flaky, but this can be unavoidable where this is caused by factors beyond our control (browser, Dynamics CRM platform, test machine performance)
Some points to think about:

Some browsers are more flaky than others. My experience is that in general chrome/chromium is less flaky than some of the other choices that are available. Edge is very close to chromium. Firefox seems to be difficult to automate and tests can be very flaky.
For example the “fixture” test suite will run cleanly for chromium, but the same suite run for firefox will have a 50% failure rate (although the tests will usually pass in firefox when run individually). Firefox also seems to run tests some 50% slower than other browsers.
There are subtle differences in the way in which different browsers react to having dates typed into date and time controls. What works in chrome may not work in firefox or edge.
Consider having fewer longer tests. Best practice would be to have a single test for each feature, but we must trade this off against lengthy the setup and tear down time, so it may be better to combine several tests into one.

Using Fixtures

Playwright best practice is to use “fixtures” to orchestrate the setup and tear down of tests.
For this project these tests will be found in the “fixture” folder in the code.

interaction-page.ts — contains lower-level functions that work directly with the Interactions page.
interaction-fixture.ts — contains the fixture code (Essentially this can be viewed as creating a subclass of the “test()” method that automatically connects to CRM before starting to run your test code)
interaction-test.test.ts — Contains the top-level test methods that use the interaction fixture.

This approach allows a test to be written very succinctly: -

  /**
   * Partly setting the specific date does not set the OOB date.
   */
  test("Partial specific date not valid", async ({ interactionPage, page }) => {
      await interactionPage.setSpecificDate("12042024", undefined);
      await interactionPage.assertOOBDate(undefined, undefined);
  });

The actual test setup is done by interaction-fixture.ts before it calls the “test” function.
The “interactionPage” provides functions to set values in our Specific Date PCF, or in the linked out of the box date field. Other functions allow the expected date and time values to be asserted.

Date Locales

It turns out that one of the biggest issues with testing date controls in CRM is dealing with date locales. The date format can be set in three different places: -

The OS level regional settings (defines the format used in the HTML date input tag)
The browser locale (defines how the browser deals with dates in Typescript and JavaScript, particularly with respect to time zones)
The CRM time zone settings. (Defines how dates are displayed in the out of the box date control). The CRM time zone setting is completely independent of the browser/OS time zone.

Consider that any or all of these three settings could be set differently, and it creates an “interesting” testing scenario.
On top of this we can add daylight saving time changes, which can also have an impact, and we therefore need to test dates with and without daylight saving.
A final point to note is that it is advisable to test with dates with a day > 12 to catch any potential issues with US/non-US format dates (mm/dd/yyyy vs dd/mm/yyyy).

A final “gotcha” is with the way the PCF control framework handles dates. Say we have a date that has been input by the user and passed back to the PCF framework through getOutput(). We would expect updateView() to give us the same date (i.e. timestamp) back through a call to updateView, wouldn’t we?
Sadly this is not the case, the raw Date value (and timestamp) in updateView will have been offset by both the browser locale timezone offset and the CRM user’s timezone offset.

The tests were originally written with UK/European date format in mind and testing with US regional format settings showed that this could cause issues, so the test were updated and tested for US region formats. All tests should be regional format agnostic now, (but have only been explicitly tested for UK and US regional formats).

Other Issues

There are a number of other points to notice in these tests:

There are differences between browsers. This can affect what is displayed and the way in which tests work. Webkit in particular is problematic (and has therefore been ignored)
For the standard out of the box date/time control, the time field only appears once date has been filled.
Copilot is quite difficult when trying to run automated tests. It will occasionally randomly capture focus. The tests have been written to deal with CoPilot by turning it off, but if possible turn off here https://admin.powerplatform.microsoft.com/tenantsettings if you value your sanity.

Conclusions

Running automated UI tests is invaluable and potentially saves huge amounts of time, especially if you must test across multiple browsers, locales and security roles. (Running the entire suite of tests in multiple locales across 4 browsers takes around 13 minutes)
From a support point of view, your tests will be invaluable when the original testing team has moved on to other projects and the knowledge of “how to test” may have been lost.
UI tests are flaky, some browsers are more flaky than others.
Don’t rely on Microsoft provided out of the box controls working cleanly under automation— they may be flaky in some circumstances.
There are unexpected variations in the way that browsers behave, even with the standard CRM controls. (Looking at you firefox)
Chrome seems to be the least flaky of the browsers tested. Edge is a close second.
Some browsers are faster than others — firefox appears to lag behind Chrome and Edge in this respect with some tests taking 50% longer in Firefox.
The performance of the online CRM system in tests is such that they will often timeout. There’s a trade-off here between increasing the timeout, and the overall time taken to run tests if there are several timeout failures.
I have set my tests to use 6 workers, which seems to work for me (and maxes out my CPU). You may want to raise or lower this depending on your system. Typically running the 32 tests for chromium should take just over 5 minutes.
You should evaluate time spent automating UI tests vs what can be achieved manually (i.e. set OS timezone/ CRM timezone manually for the test account/system and re-run test suite)
The date value that your PCF control passes back to the model driven framework through getOutput() may not be the same as the value you get back through updateView().
The way that dates in PCF controls interact with the model driven framework, OS timezone, browser locale and CRM timezone is complicated. Integration testing with playwright is invaluable in helping you to understand these interactios.
Fixtures are a useful addition to allow you abstract your test logic, to make your tests clearer and to make them more self-documenting.
Playwright can be used in many different ways to run tests interactively within the VS Code test panel, from the editor or from the command line. Test can also be run from a CI/CD pipeline.
Tools such as test recorder, debugging interface, trace interface and HTML reports with screenshots and test detail are invaluable.
Testing will ultimately enhance the quality of the UI experience as you understand better how your control operates, and prevent you making assumptions about how a control will behave in different scenarios.
Writing automated tests does takes time (and money), but once done you will be saving both time and money every time you have to run the test suite.

Resources

This github project is well worth looking at for a general take on testing Dynamics CRM with playwright.

https://github.com/BakkappaN/MicrosoftD365CRMPlaywrightFramework