Do you think there would be any worthwhile gains from relaxing address-dependent load ordering, like on Alpha/AXP? Or was that just a lot of extra pain for little reward?

In my opinion, conceptualizing memory consistency in terms of acquire/release is wrong and it confuses many programmers.

One problem is that 2 kinds of ordered loads and stores are not enough, 4 kinds are needed.

Some algorithms need not only load-acquire and store-release, but also a store that is guaranteed to be executed before all subsequent stores and a load that is guaranteed to be executed after all previous loads. These properties are the opposite of those provided by load-acquire and store-release. Except for x86 where any loads and stores behave like this, the other popular ISAs do not have such loads and stores, so stronger than necessary instructions must be used, i.e. load barriers and store barriers. Moreover, the instruction that is named a load barrier in the Arm ISA is not a load barrier, but a stronger barrier, but it must be used instead of a load barrier as no better alternative exists.

Besides the fact that not only load-acquire and store-release, but also other 2 ordered loads and stores are needed, a much more serious problem is that load-acquire is not the instruction that is really needed.

I have never seen any useful algorithm where load-acquire is the correct instruction to use. In all algorithms, what you want is not an instruction, but a loop that compares memory repeatedly, waiting for some condition to be fulfilled. The 4 most frequent kinds of loops that are needed are wait-for-not-equal, wait-for-equal, wait-for-even and wait-for-zero.

These loops may execute a load-acquire, but that is not the desired behavior. What you really need are 2 kinds of barriers, one inside the loop and one immediately after the loop.

The internal barrier must prevent the CPU from speculatively executing many future loop instances beyond the conditional jump that terminates the loop body, as it normally does. On x86-64, the instruction PAUSE provides such a barrier. On Aarch64, I suspect that a load-acquire instruction does inhibit this kind of speculative execution, despite the fact that this behavior is not documented. Otherwise, a CPU executing this kind of loop would waste a lot of energy and resources.

The barrier after such a loop must prevent speculative memory accesses beyond it. The semantics of load-acquire are not really needed, because all such loads are done in a loop and the memory accesses that follow the loop cannot be executed before such a load, due to the control dependency created by the conditional jump that follows the load. Nonetheless, while normal execution is impossible, the CPU can execute speculatively any loads following the loop and only this speculative execution can break the acquire semantics. Therefore what you really need is a speculation barrier after the loop, not an acquire barrier, whose behavior is provided automatically by the loop, even without any instruction with acquire semantics.

On x86-64, if a loop is terminated by an unconditional jump, it is said that such a jump blocks the speculative accesses beyond it. This is why the examples provided by Intel in its optimization manual about how to write this kind of acquire loop show loops terminated with unconditional jumps, even if this makes the loops longer, as otherwise the unconditional jump could have been eliminated by moving the conditional jump at the end of the loop. On x86-64, an alternative to unconditional jumps is the LFENCE instruction, which is a barrier for speculative memory accesses.

The so-called load-acquire instruction of Arm might also implement the textbook behavior of load-acquire, of ordering the memory accesses, despite the fact that this behavior is always superfluous, but it must also have the undocumented behavior of being a speculation barrier for memory accesses, otherwise the Arm CPUs would have been very inefficient. However, dedicated speculation barriers of the 2 kinds needed inside the loop and outside the loop would have been more efficient than a load-acquire instruction.

Does any other architecture allow split cacheline atomic operations across sockets?

Is this just a question of what one considers to be “significant”? I’d consider 3% to be significant but the authors apparently don’t.

It’s a question of how much resources to allocate to the hardware team, and how much resources to be distributed diffusely to the software engineers but especially to the compiler team.

Even your linked paper contends that the actual observed slowdown is as much as 22% in the Geekbench example, but the thesis is that the slowdown is not inherent to TSO, but merely to the specific hardware implementation. Is it worthwhile for a company to optimize its TSO to chase the final gains, or is it better not to have this feature in the first place and just change the compiler?

Indeed my instinct is that it is better to do this in software, where the programmer clearly communicates which stores are ordered, and which may happen in arbitrary order.

> Why doesnt Stream require their binaries to be compiled to some bytecode and transpiled during the install ?

They still have to support the entire back-catalog. It's not reasonable to expect thousands of existing games to port to ARM

I don't know, but assumed that Valve doesn't require studios to recompile their software or use any special tooling, it's basically just packaging of existing executables. This is also why they do Windows on Linux emulation.

A few reasons:

Bytecode can't really abstract the differences in memory model between the two different processors without some kind of consequence. (IE, it would be slower.) I've personally done some high performance multithreaded programming in C# / .Net, but it only "works" because C# / .Net assumes the TSO memory model. (Described in TFA.)

In contrast, games need to squeak every cycle of performance out of their chips, and optimizations can be very CPU specific. When games target bytecode, they either won't be able to take full advantage of the hardware, or otherwise will need a lot of platform-specific fallbacks (that negate the point of bytecode anyway.)

(This is why I prefer console gaming or "simple" games that don't tax the hardware.)

Existing games will not be recompiled for the new bytecode target, and they want all of those games to work regardless.

That dream of write-once-run-anywhere has been attempted for decades and is still a massive struggle. And Steam isn’t in a position to mandate that kind of massive change. They’re big, but they still have competition from other stores and from direct sales.

They’ve been through this a few times before, and they absolutely know how important it is.

If they had tried to move to Apple Silicon and said “but none of your old software will work“ it would’ve been dead in the water. Look at how well the early Windows on ARM efforts went, although they were also hamstrung by hardware.

Arm was never modeless. Thumb is a separate encoding with different instruction semantics and Jazelle ran Java bytecode. Both of them need a special branch instruction to enter. What they don't have is legacy modes like real, v8086 or native 16/32 protected that have no reason to exist when a CPU in long mode can run 16 and 32 bit code (in compatibility sub mode) just fine.

Yeah, I thought that was interesting. Knowing Apple they must’ve profiled a ton of code and decided the hit from not “fixing“ that wasn’t worth enough.

The M chips were already so much faster then the Intel chips Apple was using before (except on Mac Pro maybe) that it was probably still a net win.