fix: updating the test extension logic by shawkins · Pull Request #48759 · keycloak/keycloak

shawkins · 2026-05-06T13:46:41Z

also minor changes to the cluster config dist test to prevent reaugmentations

keycloak-github-bot · 2026-05-07T02:57:30Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.forms.LoginTest#loginDifferentUserAfterDisabledUserThrownOut

Keycloak CI - Forms IT (firefox)

org.openqa.selenium.TimeoutException: 
Navigation timed out after 10000 ms
Build info: version: '4.28.1', revision: '73f5ad48a2'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.17.0-1010-azure', java.version: '25.0.2'
Driver info: org.openqa.selenium.firefox.FirefoxDriver
...

Report flaky test

keycloak-github-bot

Unreported flaky test detected, please review

shawkins · 2026-05-07T03:17:29Z

@michalvavrik @vmuzikar I took the changes mentioned in the issue a little further by adding in the pre-initialized db reuse mentioned elsewhere. Rather than baking that init into the distribution (which we may not want to do - at least in the default location), I'm having the Raw logic create it. This does mean there will be an initialization with ever non ReInstall.NEVER test class - including DB tests that don't need the h2 database. In any case I'm mostly looking to show what the effect of the optimization is by comparing to a run with that removed as well.

I'm not sure if we really need the ReInstall.NEVER, or the reinstall enum, moving forward. We should probably just create the raw dist once, rather than for every test class.

Other than the minor changes to ClusterConfigDistTest, there's no optimizations yet for reaugmentation.

michalvavrik

Changes LGTM, test failures are related but that is probably expected.

michalvavrik · 2026-05-07T08:31:19Z

        keycloakSessionFactory.init();
+
+        // for testing, allow the quarkus application to exit early
+        if (io.quarkus.runtime.Application.currentApplication() != null && Boolean.getBoolean("init_db_only")) {


Now that we have secret system property that stops Keycloak from starting, it is all based on hope that noone ever sets init_db_only in production for their own purpose. I think there should be some logging if this happens for the unlikely situation when someone else but our tests set it.

This can be viewed as temporary - we'll refine it based upon #48733 - and yes it can have logging or a more complicated key so that it's not accidentally used.

michalvavrik · 2026-05-07T08:34:33Z

 @RawDistOnly(reason = "Requires creating JSON file to be available between containers")
 public class UpdateCommandDistTest {

+    private static final String CONFIG_FILE_HASH = "Czh0QKyu1fHcvW0XFuYb/Mdkya0ASlObok+xaRqaP8k=";


I think this will break if the default file hash changes. Also there is a test failure in CI UpdateCommandDistTest.testCacheLocalChange:157 expected: <Czh0QKyu1fHcvW0XFuYb/Mdkya0ASlObok+xaRqaP8k=> but was: <not_found>. Maybe it is expected as this is PR is marked as draft.

It's a fluke of the raw dist logic that the config file was getting deleted, so I was suprised to see that the test logic was dependent on that. This was the quick attempt at making use the actual state, but yes to be resliant it would need to compute hash, rather than just using a hardcoded one. For now I've switched the logic back to deleting the config file.

shawkins · 2026-05-07T10:42:56Z

@michalvavrik Thanks for the review. I won't address most of the comments yet as this is still in a rough draft state - I really had only intended to use it for comparison purposes on strategies for reusing the raw distribution more. In particular reusing the pre-initializated database on a per class basis as shown here does is not the correct approach as it's forcing the initialization for a number of test classes in the fast tests that did not previously need it.

My next iteration will be to remove the NEVER mode to see if we can get this done to only a single installation of the raw distribution.

shawkins · 2026-05-07T14:04:05Z

My next iteration will be to remove the NEVER mode to see if we can get this done to only a single installation of the raw distribution.

I think these changes are representative and the next intended state. There is only a single raw distribution ever created.

It seemed appropriate to use the QuarkusTestResourceLifecycleManager for this, but that also requires adding our own support for that in CLITestExtension. Maybe the quarkus team could make that easier, or we could forgo using the QuarkusTestResourceLifecycleManager and mimic the support directly in the CLITestExtension.

We create initial backups and a pre-initialized h2 at install time, then copy those in later as needed.

There is just a single reinstall mode "DEFAULT" which means reset the augmentation inbetween classes, and reset the reset of the state inbetween each test. This roughly replaces the previous BEFORE_ALL mode. NEVER and BEFORE_TEST are not something we should strive to keep - NEVER makes running an individual method problematic, and BEFORE_TEST is far too expensive.

Should be ready for review. In general these changes are a good improvement on the "slow" tests, while not impacting the "fast" tests.

shawkins · 2026-05-07T19:38:05Z

@vmuzikar updated based upon the review comments

after this and the aligment with the pr from @michalvavrik we can explore the other topics from the quick test issue - further optimization around reaugmentation beyond just dryrun, and AOT (which could be utilized via the process being used to create the pre-initialized h2 db).

Other avenues:

see how much overhead there is in things out of process and what tests would be applicable for embedded
as mentioned in Improve ClusterConfigDistTest #48754 (comment) see if things like cluster config dist test could be tested in a more targeted way

also optimized ClusterConfigDistTest closes: keycloak#48754 Signed-off-by: Steve Hawkins <shawkins@redhat.com>

keycloak-github-bot

Unreported flaky test detected, please review

keycloak-github-bot · 2026-05-07T21:31:56Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.forms.LoginTest#loginDifferentUserAfterDisabledUserThrownOut

Keycloak CI - Forms IT (firefox)

org.openqa.selenium.TimeoutException: 
Navigation timed out after 10000 ms
Build info: version: '4.28.1', revision: '73f5ad48a2'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.17.0-1010-azure', java.version: '25.0.2'
Driver info: org.openqa.selenium.firefox.FirefoxDriver
...

Report flaky test

michalvavrik

LGTM

shawkins · 2026-05-08T16:52:15Z

@pruivo @vmuzikar @michalvavrik here's a preview of using aot compilation: https://github.com/shawkins/keycloak/tree/iss48754-b built on the changes in this pr. "Slow" tests when from 33 minutes to 22. "Fast" tests went from 36 to 29. The help command dist test is failing, but I believe that is primarily due to the additional message in the output about using the additional java options. It would probably be a good idea to move the help command dist tests out of the integration tests as they do not need a running server either.

shawkins force-pushed the iss48754 branch from 3536b39 to a3d9c65 Compare May 6, 2026 14:56

shawkins changed the title ~~fix: correcting the test extension logic~~ fix: updating the test extension logic May 6, 2026

shawkins force-pushed the iss48754 branch 2 times, most recently from de1f157 to 139a399 Compare May 6, 2026 18:31

keycloak-github-bot Bot added the flaky-test label May 7, 2026

keycloak-github-bot Bot mentioned this pull request May 7, 2026

Flaky test: org.keycloak.testsuite.forms.RPInitiatedFrontChannelLogoutTest#testFrontChannelLogoutWithoutSessionRequired #48647

Open

keycloak-github-bot Bot reviewed May 7, 2026

View reviewed changes

shawkins force-pushed the iss48754 branch 2 times, most recently from 0d0fb84 to e080679 Compare May 7, 2026 03:08

michalvavrik reviewed May 7, 2026

View reviewed changes

shawkins force-pushed the iss48754 branch from 8a2e23a to f42c742 Compare May 7, 2026 13:53

shawkins marked this pull request as ready for review May 7, 2026 14:43

shawkins requested review from a team as code owners May 7, 2026 14:43

keycloak-github-bot Bot added the team/cloud-native label May 7, 2026

shawkins mentioned this pull request May 7, 2026

Quarkus IT execution time savings: skip realm bootstrap when not required #48809

Open

vmuzikar reviewed May 7, 2026

View reviewed changes

Comment thread quarkus/tests/junit5/src/main/java/org/keycloak/it/utils/RawKeycloakDistribution.java Outdated

Comment thread quarkus/tests/junit5/src/main/java/org/keycloak/it/junit5/extension/DistributionTest.java Outdated

vmuzikar requested a review from michalvavrik May 7, 2026 18:30

shawkins force-pushed the iss48754 branch from f42c742 to 92ed478 Compare May 7, 2026 19:31

fix: simplifying the management of the raw distribution

eb6c935

also optimized ClusterConfigDistTest closes: keycloak#48754 Signed-off-by: Steve Hawkins <shawkins@redhat.com>

shawkins force-pushed the iss48754 branch from 92ed478 to eb6c935 Compare May 7, 2026 19:40

keycloak-github-bot Bot reviewed May 7, 2026

View reviewed changes

michalvavrik approved these changes May 8, 2026

View reviewed changes

Conversation

shawkins commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

keycloak-github-bot Bot commented May 7, 2026

Unreported flaky test detected

org.keycloak.testsuite.forms.LoginTest#loginDifferentUserAfterDisabledUserThrownOut

Uh oh!

keycloak-github-bot Bot left a comment

Choose a reason for hiding this comment

Uh oh!

shawkins commented May 7, 2026

Uh oh!

michalvavrik left a comment

Choose a reason for hiding this comment

Uh oh!

michalvavrik May 7, 2026

Choose a reason for hiding this comment

Uh oh!

shawkins May 7, 2026

Choose a reason for hiding this comment

Uh oh!

michalvavrik May 7, 2026

Choose a reason for hiding this comment

Uh oh!

shawkins May 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

shawkins commented May 7, 2026

Uh oh!

shawkins commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shawkins commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

keycloak-github-bot Bot left a comment

Choose a reason for hiding this comment

Uh oh!

keycloak-github-bot Bot commented May 7, 2026

Unreported flaky test detected

org.keycloak.testsuite.forms.LoginTest#loginDifferentUserAfterDisabledUserThrownOut

Uh oh!

michalvavrik left a comment

Choose a reason for hiding this comment

Uh oh!

shawkins commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shawkins commented May 6, 2026 •

edited

Loading

shawkins commented May 7, 2026 •

edited

Loading

shawkins commented May 7, 2026 •

edited

Loading