Possible missing compiler memory barriers in Cortex-M BASEPRI port: TX_NOT_INTERRUPTABLE tx_queue_send can resume receiver without delivered message

Summary

TX_PORT_BASEPRI path does not appear to include a memory barrier for the compiler like the previous PRIMASK path used to include.

I am seeing what looks like a ThreadX queue/resume race when TX_NOT_INTERRUPTABLE is enabled. I believe TX_NOT_INTERRUPTABLE makes the memory barrier issue easier to surface.

Under load, a thread blocked in tx_queue_receive(..., TX_WAIT_FOREVER) can resume even though no message was actually delivered into its receive buffer, and the message that the sender attempted to hand off never appears in the queue. In my application this causes the sender-owned block to leak, because cleanup depends on either successful dequeue by the receiver or immediate send failure handling by the sender.

The most reliable trigger in my system is CDC ACM traffic after a host handshake, but the failure itself looks like a generic ThreadX queue semantic failure rather than a USBX-specific problem. I do not yet have a standalone minimal reproducer outside the application.

Update / Likely Root Cause

I now suspect the deeper root cause is not tx_queue_send alone, but the Cortex-M GNU TX_PORT_USE_BASEPRI critical-section implementation.

In the Cortex-M33 GNU port, the PRIMASK path uses inline asm with a "memory" clobber, but the BASEPRI path writes BASEPRI without an equivalent compiler memory barrier. That means enabling TX_PORT_USE_BASEPRI may allow the compiler to reorder ordinary memory accesses across TX_DISABLE / TX_RESTORE, especially under -O3 and -flto.

I locally reintroduced compiler memory barriers around the BASEPRI critical-section path and:

the queue/message-loss issue disappeared;
the application image became slightly smaller rather than larger.

Based on this, I now believe the queue symptom is likely an effect of missing compiler barriers in the BASEPRI port, with TX_NOT_INTERRUPTABLE making the problem visible.

Environment

ThreadX 6.4.0
USBX 6.4.0
STM32H563RGVx / Cortex-M33
GNU Arm Embedded toolchain
-O3
-flto
Non-secure build
TX_NOT_INTERRUPTABLE
TX_PORT_USE_BASEPRI
TX_PORT_BASEPRI=16 (corresponds to BASEPRI of 1 on system that implements 4 priority bits)

Steps To Reproduce

My real application setup is:

One thread blocks on tx_queue_receive(&queue, &msg, TX_WAIT_FOREVER).
Multiple contexts repeatedly call tx_queue_send(&queue, &msg, TX_NO_WAIT).
The most reliable trigger is traffic originating from the USBX CDC ACM read callback along with another thread after a host handshake.
After the host connects and sends a handshake command, the device starts registering outbound traffic, including realtime telemetry messages, and the failure becomes reproducible.
It is the realtime telemetry message that gets dropped

I realize this is not yet a minimal reproducer, but the observed failure mode appears consistent.

Actual Result

With TX_NOT_INTERRUPTABLE enabled:

the waiting thread can resume without a real message being placed into its receive buffer;
the message that was supposed to be delivered never appears in the queue;
the sender-side message block is leaked because the sender believes ownership was handed off;
disabling TX_NOT_INTERRUPTABLE makes the problem disappear in the same application;
enabling LTO makes the problem happen more often.
The number of spurious thread resumes corresponds 1 to 1 with the number of messages leaked

Expected Result

If tx_queue_send resumes a suspended receiver, the receiver should always observe the message that was sent.
If the message is not delivered, the sender should not observe a successful handoff.
A blocked receiver should not be resumed spuriously in a way that loses the message.

Evidence And Invariants

Application-side evidence from this project:

The leaked blocks come from a block pool
The leaked block type is typically a block intended for an outbound realtime telemetry
These leaked outbound blocks are never stamped with transmission metadata, which is only written after successful dequeue
That strongly suggests the leak happens before the suspended thread actually dequeues and processes the outbound message.
The sender failure path releases the block immediately on non-success from tx_queue_send, so the leak implies the send side believed the handoff succeeded.
The most reliable trigger path starts from the CDC ACM read callback
Other threads using similar application patterns appear healthy in the same firmware, including other threads that block on queues
Locally restoring compiler memory barriers in the BASEPRI critical-section path makes the issue disappear in the same firmware/configuration.

Interrupt/preemption notes:

This build uses TX_PORT_USE_BASEPRI with TX_PORT_BASEPRI=16.
In this configuration, only one interrupt remains above the ThreadX BASEPRI mask in my project
That interrupt does not call ThreadX services.

Suspected Area / Related Report

I originally suspected the TX_NOT_INTERRUPTABLE fast path in tx_queue_send, specifically the empty-queue / suspended-receiver handoff path. I now think that path may only be where the failure becomes visible.

The stronger suspect is the Cortex-M GNU TX_PORT_USE_BASEPRI port implementation, where the BASEPRI interrupt disable/restore helpers appear to lack the compiler memory barrier semantics present in the PRIMASK path. If so, normal memory operations in ThreadX critical sections may be reordered across TX_DISABLE / TX_RESTORE under optimization/LTO.

In ThreadX 6.4.0, Middlewares/ST/threadx/common/src/tx_queue_send.c has a path where it:

selects the suspended thread,
unlinks it from the queue suspension list,
clears the suspend cleanup pointer,
copies the message into thread_ptr->tx_thread_additional_suspend_info,
sets tx_thread_suspend_status = TX_SUCCESS,
then calls _tx_thread_system_ni_resume(thread_ptr) while interrupts are still disabled.
This may also be related to usbx in the sense that usb interrupts that modify kernel state are interfering with threadx services.

I am not claiming this is definitely the root cause, but this is the area that currently looks most suspicious.

This may be related to threadx#258, which also describes a suspended thread resuming from tx_queue_receive with an effectively empty message under load. I am only linking it as possibly related, not as proof of the same root cause.

With the barrier restored, the application image is slightly smaller, so the fix does not appear to be introducing a size regression in this build.

Questions For Maintainers

Is this a known issue in the TX_NOT_INTERRUPTABLE queue/resume path?
Is calling tx_queue_send(..., TX_NO_WAIT) from the USBX CDC ACM read-callback path an unsupported context assumption that could explain this behavior?
Does usbx code that runs from interrupt context modify kernel state in such a way to interfere with other threadx services?
If that callback context is supported, does the behavior above match any known race involving _tx_queue_send and _tx_thread_system_ni_resume?
Is there a recommended diagnostic or patch I should try next to confirm whether this is a ThreadX kernel bug?
Is the absence of a compiler memory barrier in the Cortex-M GNU TX_PORT_USE_BASEPRI path intentional?
Should the BASEPRI path provide the same compiler barrier semantics as the PRIMASK path for TX_DISABLE / TX_RESTORE?

I may not have set the appropriate labels, so apologies for that.

@Kairalite

Hi @Kairalite.

No worries about the labels; I applied the correct ones.

I will ask an experienced team member to have a look and get back to you.

@Kairalite

I took the time to look into this. This could be caused by missing compiler memory barriers in the inline assembly implementation of interrupt control macros when TX_PORT_USE_BASEPRI is enabled.

In the Cortex-M ports (specifically Cortex-M3, M4, M7, M23, M33, etc.), the interrupt disable and restore macros are implemented using inline functions that wrap assembly instructions. When TX_PORT_USE_BASEPRI is defined, these functions use MSR BASEPRI to manage interrupt masking.

In ports/cortex_m33/gnu/inc/tx_port.h (and similar files for other M-profile cores):

 1 __attribute__( ( always_inline ) ) static inline void __set_basepri_value(UINT basepri_value)
 2 {
 3     __asm__ volatile ("MSR  BASEPRI,%0 ": : "r" (basepri_value)); // Missing "memory" clobber!
 4 }

In contrast, the PRIMASK path correctly includes the "memory" clobber:

1 __asm__ volatile ("MSR  PRIMASK,%0": : "r" (int_posture): "memory");

The "memory" clobber serves as a compiler barrier. It tells the compiler that the assembly block may read or write to any memory location, preventing the compiler from reordering memory accesses across the assembly instruction.

Without this barrier:

The compiler (especially with -O3 and -flto) may move memory reads or writes from inside a critical section to outside (either before the disable or after the restore).
It may also move the interrupt enable/restore instruction up into the middle of the code it was supposed to protect.

This issue appears to affect multiple Cortex-M port variants (GNU, AC6, and IAR) that utilise the unified inline versions of interrupt control macros and support the TX_PORT_USE_BASEPRI configuration.

@Kairalite Can you try to implement a fix as described above and report your findings, please?

@fdesbiens

@fdesbiens

Thank you for getting back to me, so a few findings.

with TX_NOT_INTERRUPTABLE enabled
Adding the memory barrier back in, does appear to resolve the issue, but only partially.
Instead of consistently dropping tx_queue messages, it will only occasionally drop tx_queue messages (and spuriously resume the suspended thread). This is also sensitive to other firmware behaviors around threadx services (rate of message drop between almost 0 to 25% of the time).

Another thing worth noting, when the thread is spuriously resumed, the tx_queue_receive does NOT return TX_SUCCESS.

with TX_NOT_INTERRUPTABLE disabled
Adding the memory barrier resolves the issue entirely, no messages are dropped.
Without the memory barrier, I still have occasional queue message drops and spurious thread resumes but at a much lower rate than with TX_NOT_INTERRUPTABLE enabled.

So, I suspect there may also be issues with the TX_NOT_INTERRUPTABLE code paths as well, but I understand that will require a very rigorous and time-consuming analysis, especially since I am using very aggressive optimization settings.

I will build without TX_NOT_INTERRUPTABLE for now.

For the time being, could you plan to patch the missing memory barriers in the TX_PORT_USE_BASEPRI paths?

Thank you so much!

After narrowing this down further, I think there are two separate ordering bugs involved on Cortex-M33/GNU when TX_PORT_USE_BASEPRI and TX_NOT_INTERRUPTABLE are enabled.

There is a compiler-ordering problem in the BASEPRI critical-section path. A compiler barrier is still needed so queue/suspend bookkeeping, direct queue message copies, and tx_thread_suspend_status updates are not reordered too aggressively under -O3 -flto.
There is also an architectural handoff problem in the non-interruptible scheduler return path. In the Cortex-M33 GNU port, _tx_thread_system_return_inline() and _tx_thread_system_return() set PendSV and then briefly unmask interrupts, but without DSB/ISB around that handoff. In the TX_NOT_INTERRUPTABLE suspend/resume paths, that means the current thread can fall through past the logical suspend point before PendSV is actually taken. The observable symptom on my side was that tx_queue_receive(..., TX_WAIT_FOREVER) could resume spuriously without returning TX_SUCCESS, and a later direct queue handoff would then appear to be lost.

Locally, the combination that fixed it was:

compiler fences for the BASEPRI path
DSB/ISB in the Cortex-M33 GNU _tx_thread_system_return_inline() / _tx_thread_system_return() PendSV handoff path

TX_INLINE_THREAD_RESUME_SUSPEND looks like the same class of risk if enabled, because it creates another inlined scheduler handoff boundary. By contrast, this does not look like a generic queue algorithm bug to me; it looks like a missing ordering/synchronization issue that becomes visible specifically in the NI handoff path.

The small window on PendSV triggering explains why the message drop + spurious resume is intermittent rather than consistent, I suspect both of these had a factor in the issue here, as without the memory barrier it is consistent.

Fix race condition and message loss in Cortex-M GNU ports (eclipse-th…

Fix race condition and message loss in Cortex-M AC6 ports (eclipse-th…

Fix race condition and message loss in Cortex-M IAR ports (eclipse-th…

Fix race condition and message loss in additional Cortex-M AC6 ports (e…

Fix race condition and message loss in all remaining Cortex-M IAR por…

@Kairalite

@Kairalite, I put together a PR to address the issue across our whole set of Cortex-M ports.

#523

Please have a look.

@fdesbiens

@fdesbiens
Thank you!

Added a couple suggestions, mainly ISB after restoring interrupts since we expect PendSV to be pending and want it to trigger immediately.

Pulling in same ARM citations:
as per Arm DAI 0321A PDF

“If it is necessary to ensure a pended interrupt is recognized before subsequent operations, the ISB instruction should be used after CPSIE I.”
(section 4.7, p.28; lines 1077-1080)

__enable_irq(); // CPSIE I : Enable interrupt __ISB(); // Allow pended interrupts to be recognized
(p.28; lines 1083-1087)

“you might want to insert an ISB instruction if the priority level change can result in the interrupt being accepted, and you want this interrupt to be executed immediately.”
(p.34; lines 1321-1324)

Added ISB barriers to scheduler return paths in Cortex-M GNU ports (e…

common/tx_queue_cleanup: document why NI path omits revalidation guards

Fixed race condition and message loss in Cortex-M ports (#523)

Merged PR #523 which addresses the issues in this ticket.

Navigation Menu

Repository navigation

Possible missing compiler memory barriers in Cortex-M BASEPRI port: TX_NOT_INTERRUPTABLE tx_queue_send can resume receiver without delivered message #516

Summary

Update / Likely Root Cause

Environment

Steps To Reproduce

Actual Result

Expected Result

Evidence And Invariants

Suspected Area / Related Report

Questions For Maintainers

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Possible missing compiler memory barriers in Cortex-M BASEPRI port: TX_NOT_INTERRUPTABLE tx_queue_send can resume receiver without delivered message #516

Description

Summary

Update / Likely Root Cause

Environment

Steps To Reproduce

Actual Result

Expected Result

Evidence And Invariants

Suspected Area / Related Report

Questions For Maintainers

Activity

Kairalite commented on Mar 23, 2026

fdesbiens commented on Mar 25, 2026

fdesbiens commented on Apr 10, 2026

Kairalite commented on Apr 11, 2026

Kairalite commented on Apr 11, 2026

fdesbiens commented on Apr 15, 2026

Kairalite commented on Apr 15, 2026

fdesbiens commented on Apr 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions