Repository navigation
profiling.sampling CPU mode may attribute on-CPU samples to detached tstates when sharing native_thread_id #159312
Description
Bug report
My reproducer uses one worker OS thread. The worker enters funA -> funB using tstate0, detaches tstate0, creates and attaches tstate1 in the same interpreter, then runs funC -> funD. Only funD performs CPU work. Despite that, CPU-mode output attributes about half of the direct samples to the suspended funB/tstate0 stack and half to the executing funD/tstate1 stack.
import os
import sys
import threading
import _native
def funD():
print(
f"HOT LOOP funD pid={os.getpid()} tid={threading.get_native_id()}",
file=sys.stderr,
flush=True,
)
value = 1
while True:
value = (value * 1_103_515_245 + 12_345) & 0x7FFF_FFFF
value ^= value >> 7
def funC():
_native.print_tstate("tstate1 in funC")
funD()
def funB():
_native.switch_tstate_and_call(funC)
def funA():
funB()
def work0():
print(
f"WORKER tid={threading.get_native_id()}",
file=sys.stderr,
flush=True,
)
_native.print_tstate("initial tstate0")
funA()
def main():
print(
f"MAIN tid={threading.get_native_id()}",
file=sys.stderr,
flush=True,
)
worker = threading.Thread(target=work0, name="work0")
worker.start()
worker.join()
if __name__ == "__main__":
main()#define _GNU_SOURCE
#include <Python.h>
#include <stdio.h>
#include <sys/syscall.h>
#include <unistd.h>
static long
gettid_linux(void)
{
return (long)syscall(SYS_gettid);
}
static void
print_tstate_info(const char *where, PyThreadState *tstate)
{
PyThreadState *current = PyThreadState_GetUnchecked();
unsigned long native_tid = tstate ? tstate->native_thread_id : 0;
long tid = gettid_linux();
fprintf(stderr,
"TSTATE %-24s tstate=%p current=%p native_thread_id=%lu "
"gettid=%ld match=%s\n",
where, (void *)tstate, (void *)current, native_tid, tid,
tstate && native_tid == (unsigned long)tid ? "yes" : "no");
fflush(stderr);
}
static PyObject *
print_tstate(PyObject *self, PyObject *args)
{
const char *where;
if (!PyArg_ParseTuple(args, "s", &where)) {
return NULL;
}
print_tstate_info(where, PyThreadState_Get());
Py_RETURN_NONE;
}
static PyObject *
switch_tstate_and_call(PyObject *self, PyObject *callback)
{
if (!PyCallable_Check(callback)) {
PyErr_SetString(PyExc_TypeError, "callback must be callable");
return NULL;
}
PyThreadState *tstate0 = PyThreadState_Get();
PyInterpreterState *interp = PyThreadState_GetInterpreter(tstate0);
print_tstate_info("tstate0 in funB", tstate0);
PyThreadState *saved0 = PyEval_SaveThread();
print_tstate_info("tstate0 detached", saved0);
PyThreadState *tstate1 = PyThreadState_New(interp);
if (tstate1 == NULL) {
PyEval_RestoreThread(saved0);
return PyErr_NoMemory();
}
print_tstate_info("tstate1 created", tstate1);
PyEval_RestoreThread(tstate1);
print_tstate_info("tstate1 attached", tstate1);
/* The callback enters funC -> funD, and funD intentionally does not return. */
PyObject *result = PyObject_CallNoArgs(callback);
/* Cleanup remains correct if funC is changed to return. */
PyObject *raised = result ? NULL : PyErr_GetRaisedException();
Py_XDECREF(result);
PyThreadState *saved1 = PyEval_SaveThread();
PyEval_RestoreThread(saved0);
PyThreadState_Clear(saved1);
PyThreadState_Delete(saved1);
if (raised) {
PyErr_SetRaisedException(raised);
return NULL;
}
Py_RETURN_NONE;
}
static PyMethodDef methods[] = {
{"print_tstate", print_tstate, METH_VARARGS, NULL},
{"switch_tstate_and_call", switch_tstate_and_call, METH_O, NULL},
{NULL, NULL, 0, NULL}
};
static PyModuleDef_Slot slots[] = {
{Py_mod_multiple_interpreters, Py_MOD_PER_INTERPRETER_GIL_SUPPORTED},
{Py_mod_gil, Py_MOD_GIL_NOT_USED},
{0, NULL}
};
static struct PyModuleDef module = {
PyModuleDef_HEAD_INIT, "_native", NULL, 0, methods, slots
};
PyMODINIT_FUNC
PyInit__native(void)
{
return PyModuleDef_Init(&module);
}and the sample result is:
cpython315-tstate-tachyon-demo.zip
CPython versions tested on:
3.15
Operating systems tested on:
Linux
Activity
maurycy commentedon Oct 13, 2026
@pablogsal Maybe we should check
tstate_status.bound_gilstatebefore settingTHREAD_STATUS_ON_CPU?pablogsal commentedon Oct 13, 2026
We should probably fix this, but what's the use case for creating a second thread state on the same OS thread in the same interpreter? Did you run into this in an actual application, or is this something you found while experimenting?
pythongh-159312: Confine CPU attribution checks to unbound states
superajun-wsj commentedon Oct 14, 2026
Hi @pablogsal，Since Python 3.15 was released only recently, we haven’t yet used Tachyon with 3.15 in a production workload. This report is based on the targeted reproducer above. The association between a PyThreadState and a native thread can change in CPython.
pablogsal commentedon Oct 14, 2026
Yeah but how you arrived to this reproducer?
superajun-wsj commentedon Oct 14, 2026
Since Tachyon was introduced, we’ve been interested in using it. We’d like to find a way to extend it to support mixed Python/native stack traces, which it does not currently provide.
While studying Tachyon’s implementation, we noticed that its on-CPU mode reconstructs Python call stacks starting from a PyThreadState. In CPython, the association between a thread state and an OS thread can change over time. That led us to suspect that samples might be attributed to the wrong thread state. We constructed the reproducer in this issue to test that hypothesis, and it reproduced the problem. @liuzhijie-0614
liuzhijie-0614 commentedon Oct 15, 2026
One tricky case is when A calls native code that switches to B. After B is unbound, the thread may still be doing native work on behalf of A, but A’s bound_gilstate flag is not automatically restored.
Filtering solely on bound_gilstate could miss these CPU samples, while falling back to all unbound stacks could double-count them. @maurycy @pablogsal , do you have any suggestions for handling this?
pythongh-159312: Resolve CPU attribution from collected stacks
pablogsal commentedon Oct 15, 2026
I'm still thinking about this, but I don't currently see it as a practical problem. I'm not aware of real software using this pattern, and I don't understand why an application would need it. py-spy and Austin have the same issue.
We'll keep exploring a cheap fix, but I'm not willing to add an expensive check just to cover this case.
pablogsal commentedon Oct 15, 2026
I'll soon open source a version of Tachyon that I built at my current employer. It supports mixed Python/native stacks and much more 🙂
superajun-wsj commentedon Oct 15, 2026
That's great news! Thank you so much for your work on this and for sharing it with the community. I'm really looking forward to the open-source release, especially the mixed Python/native stack support!
superajun-wsj commentedon Oct 15, 2026
Ok, thank you very much!
maurycy commentedon Oct 15, 2026
@pablogsal (Maybe to state the obvious) To me this problem is similar to #159259 (comment). In both cases, TID alone is not sufficient to uniquely identify
PyThreadState. I don't know if there's a comprehensive solution that doesn't involve changing the interpreter, and I don't see any lightweight way (w/o crazy_PyEval_EvalFrame()bookkeeping) to approach this, but I didn't think too hard.liuzhijie-0614 commentedon Oct 16, 2026
Yeah, I was thinking along similar lines: tracking an “owner” tstate per OS thread, saving the previous owner (A) before entering _PyEval_EvalFrame() for B, and restoring A on return. But that wouldn’t quite align with the period during which the tstate is bound to the OS thread, would still require interpreter changes, and would be a bit more involved.
pablogsal commentedon Oct 16, 2026
@liuzhijie-0614
Yeah, I explored this too. My concern is adding work on every eval loop entry and exit just to cover this case. I'd want the bookkeeping to happen only when thread states switch. I haven't found an approach that handles all the cases correctly yet.
pablogsal commentedon Oct 16, 2026
@maurycy
Yeah, although exposing anything other than
tidin the public interface would be a bad idea. We should handle thread-state identity internally. I'm open to a small interpreter change, provided we preserve the ABI and keep the extra work out of the normal call path 🙂liuzhijie-0614 commentedon Oct 17, 2026
Make sense to me 🙂