Wine: Signature Checks and FL Studio

By Kitsune

FL Studio sent me into a part of Wine I hadn’t expected to spend the evening investigating: signature verification.

The failure I was following returned 0x80096004 from WinVerifyTrust. The investigation involved FL Studio 26.1.3 and the GE-Proton11-6/Wine environment I was testing. Before treating it as a general application compatibility problem, I wanted to understand why this particular check was failing.

That narrowed the work considerably. Instead of asking whether the entire application worked, I could compare what happened to the signed data along the verification path.

Follow the representation

The investigation pointed toward the reconstruction of CMS authenticated attributes in Wine’s crypt32 implementation.

The relevant code decoded the attributes and encoded them again before hashing. The normal encoder sorted the outer attribute set. Our working explanation was that this changed the byte representation of the particular input under investigation, whose attributes had arrived in a different order.

That distinction is easy to miss when reading structured data. A collection can still contain the same attributes after a transformation, while the bytes supplied to a signature check have changed.

For this case, the important question became whether the verifier was reconstructing the representation covered by the signature.

Same values, different hash input

The example below uses a deliberately simple text representation. Sorting preserves the values, but changes the serialized bytes and their SHA-256 digest. It doesn’t implement CMS or verify a signature.

There is an important standards limit here. CMS requires DER encoding for signed attributes, so preserving an input’s outer order is not a general replacement for canonical encoding. The patch was a compatibility experiment for the failing input, not evidence that unsorted attributes are universally correct.

A small change with a specific purpose

The patch added a helper in dlls/crypt32/msg.c to rebuild the authenticated attributes while preserving their decoded outer order.

It was applied to the verification path. The existing encoding path remained in use when creating signatures, and individual attributes still went through the normal encoder.

That scope matters. The patch continued into the normal hashing and signature-checking flow; it did not replace the result with an unconditional success value.

The concrete result from my patched test was that WinVerifyTrust returned 0x00000000. After following the failure through the compatibility layer, seeing that result was a satisfying milestone.

It also had a clear limit: this demonstrated a change in the verification result for the case I was testing. It didn’t establish that every application workflow, every signed file, or every Proton runner was now correct.

A successful test still needs a home

The next question was how to make the change persist in the environment actually running the application.

A patch tested in one Wine build does not automatically change the compatibility tool selected by a launcher. I needed to keep track of which implementation the process was loading and how the change would be packaged for repeated use.

That is particularly relevant to my work around daw-proton-ge. A locally convincing experiment needs a reproducible build and a useful regression test before it becomes a maintainable compatibility change.

We also discussed a limitation in the approach: preserving the decoded order during re-encoding is narrower than preserving the original encoded representation throughout. That remained a reason to investigate further before treating the proof of concept as a complete solution.

I don’t have an upstream merge or a finished regression-test suite to claim from this. What I do have is a specific failure, a code-level explanation supported by the experiment, and a successful patched verification result.

I like this example because the visible problem was “FL Studio doesn’t work,” while the useful question ended up being about a handful of bytes reconstructed much further down the stack.