Hide native Windows 11 Start button when using a replacement

[YellowNest]

Summary

Hide the native Windows 11 Start button visuals while Open-Shell's replacement Start button is enabled.

Windows 11 renders the Start button through XAML, so the older HWND-based hiding logic cannot remove the native glyph. This caused custom Open-Shell buttons to be drawn on top of the Windows Start icon.

The change uses the Windows XAML diagnostics visual tree to:

• collapse the native Start glyph
• disable hit testing on the underlying native Start control
• preserve the native control's layout slot
• restore the original XAML properties when replacement is disabled or Open-Shell exits
• respect the All taskbars setting

The replacement button itself is not moved or resized by this code, so Windows remains responsible for centered taskbar layout.

Verification

Tested on Windows 11 25H2 with centered taskbar:

• Custom replacement button no longer shows the Windows Start glyph underneath.
• Replacement button remains in the correct centered Start position.
• Left-click and right-click behavior continue to work.
• Disabling Replace Start button restores the native Windows Start button.
• Re-enabling it hides the native button again.
• Exiting Open-Shell restores the native Start button without restarting Explorer.

GitHub Actions Build completed successfully for the current head e9039e3:

https://github.com/Open-Shell/Open-Shell-Menu/actions/runs/37805091803

The lifecycle implementation avoids loading StartMenuHelper and initializing XAML Diagnostics when the replacement Start button is disabled.

Repeated Explorer exit/restart and complete injected DLL unloading still require runtime verification. A successful build does not establish those behaviors.

Fixes #1631

[ge0rdi]

This is super awesome.
I knew about the Explorer TAP interface from TranslucentTB project. But never managed to properly use it inside Open-Shell.

Glad to see it works nicely. And I guess it should be used for more customizations and improvements related to start button and taskbar itself.
For example there is this very ugly mouse hook HookDesktopThreadMouse that we are using to intercept mouse events that would be consumed by XAML framework otherwise. It would be great to get rid of this by changing XAML handler for start button.

Do you plan to work more in this area? I think it would be really appreciated.

[ge0rdi]

I have noticed that enabling custom Start button cause also that TaskView button dissapears. There is just empty place instead of the button.

[ge0rdi]

Another issue is that when I Exit Open-Shell (via right-click menu), it is not possible to start it again.

Starting Open-Shell Menu Settings shortcut doesn't do anything (no settings are shown, Start menu is not customized).

[YellowNest]

Thanks for testing this carefully. Both reports were valid and pointed to separate problems in the initial TAP implementation.

I pushed the follow-up in 9f572bb and ran a fresh Actions build against that exact commit; the build passes.

The Task View regression came from treating every ExperienceToggleButton#LaunchListButton as the Start control. Current Windows 11 uses that shape for multiple taskbar controls. The code now resolves the attached AutomationId and only classifies the actual StartButton for suppression, so Task View is left alone.

The restart problem was a separate TAP lifetime issue. Shutdown now restores the XAML properties synchronously, removes the visual-tree callback with UnadviseVisualTreeChange, tears down the dispatch window, and balances the additional module reference introduced by InitializeXamlDiagnosticsEx. There is also a shutdown guard preventing a late connection attempt from attaching again while Open-Shell is exiting.

The build for the follow-up is here:
https://github.com/YellowNest/Open-Shell-Menu/actions/runs/37219583485

The two paths that still deserve runtime confirmation are:

• custom Start button enabled while Task View remains visible and functional
• Exit Open-Shell, then start Open-Shell again without restarting Explorer

And yes, I plan to keep working in this area. Replacing HookDesktopThreadMouse with a narrower XAML-side solution looks like a good next target. I want to keep this PR focused and make sure the TAP lifecycle is solid first, then tackle that separately.

[ge0rdi]

I want to keep this PR focused and make sure the TAP lifecycle is solid first, then tackle that separately.

Definitely.
Each fix/feature deserves own PR.

I'm glad you plan to work on this more.

[YellowNest]

Thanks, I really appreciate that.

I use Open-Shell myself, so contributing here is useful to me too. I care about keeping it working well on current Windows builds, and if I can help take some of the load off, I'm happy to.

I also appreciate the trust you're putting in the work. Your reviews and testing are valuable, especially around these Windows 11 edge cases, so I'll keep changes focused and separate like you suggested.

I've already started looking at the XAML input side in a separate research branch. The first build is clean, but I'll keep it out of upstream until I've tested the behavior properly.

Glad to help.

[ge0rdi]

I think it is time to do step (or actually several steps) back.
Using AI is great, but you have to steer it towards reasonable code.

Right now the code is rather convoluted, full of weird complex constructs that cover edge cases that are not likely to happen.
This would be very hard to maintain in the future.

I think we should start with very simple approach that covers actual behavior of XAML Diagnostics framework.
We can then add more complex checks/behavior if needed.

I have prepared the state that I have in my mind here:
https://github.com/Open-Shell/Open-Shell-Menu/blob/simple-tap/Src/StartMenu/StartMenuHelper/Win11StartButtonTap.cpp

It is based on this PR, with greatly simplified TAP handling.
I have watched it in debugger to see how it behaves and it seems to work just fine (tested on Win11 24H2).

I'm still not 100% satisfied with it (for example those variable/file names should not refer to Win11 and Start button as discussed elsewhere). But we should start with this.

Please, if you don't mind, have a look. We can discuss any potential issues that may be still there and decide how (whether) to cover them.

I really appreciate your time (and help with this). I'm trying to be supportive here, but I'm just not fully content with the direction it went.

[ge0rdi]

It seems that we have to always connect to VisualDiagConnection1 endpoint.
Currently the connection doesn't work during boot, because we load too early and the API fails (I guess XAML is not ready at that point). Then we are trying endpoints with different numbers and the API simply always fails.

If we connect to 1 always, it will eventually succeed (usually on second attempt).

Previous code where we cycled between 1/2 worked too. Because eventually we did try 1 again.
I'm not sure why other tools try those different endpoint names.

[YellowNest]

Thanks for taking the time to review this and put together a simpler implementation.

I agree with your assessment. The current implementation has become unnecessarily complicated, and continuing in that direction would make future maintenance harder.

I'll use your simple-tap branch as the baseline going forward. Any further changes should be small, clearly justified by reproducible behavior, and tested before being proposed.

I appreciate your patience and the effort you've put into guiding this toward a cleaner solution.

[ge0rdi]

I'll use your simple-tap branch as the baseline going forward.

I have pushed few fixes to the branch.
Will also push some more simplifications (use of ATL factory) in few minutes.
So, please, wait for that (and then we can continue here).

[ge0rdi]

Ok, I have all things pushed in simple-tap branch.
Feel free to grab it from there.