Skip to content

Conversation

@icyca
Copy link

@icyca icyca commented Jan 6, 2026

Description

This PR addresses a race condition when multiple DistributedApplicationTestingBuilder instances are initialized concurrently (e.g., during parallel integration tests).
It's a pretty passive solution that uses simple mutexes to prevent that rare case.

The Problem
Previously, the logic for handling auto-generated parameters (like passwords) followed this flow:

  1. Generate a new value immediately (e.g., Thread A generates Pass1, Thread B generates Pass2).
  2. Lock the user secrets store.
  3. Save the value.

By the time the lock was acquired, the value was already generated. This caused a "last-writer-wins" scenario where:

  • Thread A persists Pass1 and starts its container.
  • Thread B overwrites secrets with Pass2.
  • Thread A (or dependent services) tries to read the secret, gets Pass2 (the wrong password), and fails authentication against the container running with Pass1.

The Solution
I've introduced a new
GetOrSetSecret
pattern that inverts this flow to be atomic. The new flow is:

  1. Lock (using a system-wide Mutex to handle cross-process locking).
  2. Reload the secrets from disk.
  3. Check if the value already exists.
  • If yes: Return the existing value (Thread B now sees Pass1 and uses it).
  • If no: Generate and Save the new value.

Implementation Details
Added
GetOrSetSecret
to
IUserSecretsManager

Implemented a Mutex based on the file path hash in
UserSecretsManagerFactory
to ensure we coordinate writes effectively across test processes, not just threads.
Added resilience: if locking fails (timeout) or file IO fails, we degrade gracefully to the generated value to prevent the test suite from crashing, though the race condition might technically persist in that specific failure edge case.
This ensures both test instances agree on the same shared secret (e.g., Redis password) and containers start/connect successfully.

Fixes #13720

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No
  • Does the change require an update in our Aspire docs?

@icyca icyca requested a review from mitchdenny as a code owner January 6, 2026 20:15
@github-actions
Copy link
Contributor

github-actions bot commented Jan 6, 2026

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 13774

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 13774"

@dotnet-policy-service dotnet-policy-service bot added the community-contribution Indicates that the PR has been added by a community member label Jan 6, 2026
@icyca
Copy link
Author

icyca commented Jan 6, 2026

@dotnet-policy-service agree

Debug.WriteLine($"Failed to set value for parameter '{parameterName}' in application '{applicationName}' to user secrets.");
}
return value;
Debug.WriteLine($"UserSecretsParameterDefault: applicationName: {applicationName}, parameterName: {parameterName}");
Copy link
Member

@JamesNK JamesNK Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this debug is appropriate anymore. It's now being written every time GetDefaultValue is called. Before it was written only on a failure.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, but i just added it in order to use applicationName somehow since there was an error raised if left unused. And i didn't wanna change the signature of UserSecretsParameterDefault . I know this may be something trivial but what would you suggest? Thanks. @JamesNK

/// <param name="name">The name of the secret.</param>
/// <param name="valueGenerator">Function to generate the value if it doesn't exist.</param>
/// <returns>The existing or generated secret value.</returns>
string GetOrSetSecret(string name, Func<string> valueGenerator);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A problem here is there isn't any indication of whether the secret was successfully set. It could be TryGetOrSetSecret and return a bool indicating whether the value was successfully set. But I've never seen the Try pattern combined with GetOrSet before.

Maybe it doesn't matter because the other thing the bool result is being used for is writing a debug message.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an internal API so we could change it in the future easily if needed. Doesn't need to be solved now.

}
}

public void GetOrSetSecret(IConfigurationManager configuration, string name, Func<string> valueGenerator)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this overload, which works very differently, would be confusing. Rename it to InitializeConfigSecret?

@JamesNK
Copy link
Member

JamesNK commented Jan 7, 2026

Thanks for the PR. I added some comments.

icyca and others added 3 commits January 7, 2026 22:10
Co-authored-by: James Newton-King <james@newtonking.com>
Co-authored-by: James Newton-King <james@newtonking.com>
Co-authored-by: James Newton-King <james@newtonking.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Indicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrency issue with password generation of persistent containers

2 participants