[Bug]: Flaky tests because of the SetTestIdAttribute

[FlorianArnould]

Version

1.57.0

Steps to reproduce

My case is a bit difficult to reproduce as it is a flaky behavior, but I will try to keep it simple.

We have a fullstack application using angular and ASP.NET Core and pre-existing e2e test using cypress.
We already used the data test id methodology but with the data-test-id attribute name instead. So we implemented the tests using Nunit and a setup method in base class for all our tests with Playwright.Selectors.SetTestIdAttribute("data-test-id");.

After multiple days of investigation, replacing all the Page.GetByTestId("XXX") by Page.Locator("[data-test-id='XXX']") solved the flakiness.

Here is a trace presenting the issue.

trace.trace.txt

In this trace you can see in the first line : "testIdAttributeName":"data-testid" which is wrong.
Then, later you can see the wrong selector requested at line 308: "selector":"internal:testid=[data-testid=\"paragraph-date\"]"
Then, the actual timeout at line 322.

During this execution I tried to investigate with this shitty code :

bool passed = false;
for (int i = 0; !passed && i < 2; i++) {
    try {
        await Page
            .GetByTestId("paragraph-date")
            .ClickAsync();
        await Page
            .GetByTestId("paragraph-date")
            .FillAsync(text);
        passed = true;
        if (i == 1) Assert.Fail("Recovering works");
    } catch (Exception e) {
        await TestContext.Out.WriteLineAsync("Shit");
        await TestContext.Out.WriteLineAsync(e.Message);
        await TestContext.Error.WriteLineAsync(e.StackTrace);
        if (i == 1) throw;
    }
}

So at line 324, you can see it retry with right selector "selector":"internal:testid=[data-test-id=\"paragraph-date\"]"

For info the implementation looks like this :

public class EndToEndTestBase : PageTest {
    [SetUp]
    public async Task BeforeEach() {
        Playwright.Selectors.SetTestIdAttribute("data-test-id");
        // ...
        await Page.GotoAsync(BaseUri.ToString());
        await Page.EvaluateAsync(
            """
            async () => {
                localStorage.setItem('cookies-consent', JSON.stringify(
                {
              "performance": false,
              "targeting": false
                }))
            }
            """
        );
    }
}

[Parallelizable(ParallelScope.Self)]
[TestFixture]
public class NewProjectForm : EndToEndTestBase {
    [Test]
    public async Task ShouldFillInForm() {
        // ...
    }
}

Expected behavior

The SetTestIdAttribute should be reliable.

Actual behavior

The SetTestIdAttribute create some flakiness

Additional context

I would like ask a question instead about this code :
https://github.com/microsoft/playwright-dotnet/blob/2373e8f692ff31f60d92a22396257fc862daffa9/src/Playwright/Core/Selectors.cs#L66C1-L79C6

    public void SetTestIdAttribute(string attributeName)
    {
        Locator.SetTestIdAttribute(attributeName);
        _testIdAttributeName = attributeName;
        foreach (var context in _contextsForSelectors.ToArray())
        {
            context.SendMessageToServerAsync(
            "setTestIdAttributeName",
            new Dictionary<string, object?>
            {
                ["testIdAttributeName"] = attributeName,
            }).IgnoreException();
        }
    }

This code seems to be fire and forget instead of async to be awaited when use, and what happens if it doesn't work ?
What could happen here, is ignore the exception a good idea ?

Environment

- Operating System: [Ubuntu 24.04]
- CPU: [amd64]
- Browser: [Chromium]
- .NET Version (TFM): [net9.0]

[FlorianArnould]

After a few hours without thinking about it, I got the feeling it could be the race condition because of the parallel tests and this static variable :

playwright-dotnet/src/Playwright/Core/Locator.cs

Line 49 in 2373e8f

private static string _testIdAttributeName = "data-testid";

Maybe slamming an AsyncLocal on it could fix this easily.