product-marketing never activated in 200 attempts, and positioning updates bypass .agents/product-marketing.md

[ktlesr]

I measured how 14 of these skills behave when installed together — which one a
request reaches, and whether the "every skill checks product-marketing first"
claim from the README holds. 200 attempts, one model, pinned; method and raw
numbers are linked at the end. Most of it is good news for the design. Two
parts are not.

What works: the skills do not fight each other

I loaded cro, signup, onboarding, popups, paywalls, copywriting,
copy-editing, emails, cold-email, seo-audit, ai-seo,
programmatic-seo, schema and product-marketing together, and wrote each
case so that its expected winner is the one your own descriptions route to
("For popup copy, see popups", "Distinct from public pricing pages (see cro)").
No prompt named a skill.

Of 50 activations, 49 reached the skill the descriptions point to. The one
leak was a newsletter form that reached signup once in ten, although
signup's description sends lead-capture forms to cro. The cross-references
in the descriptions work.

1. product-marketing never activates, and the context file goes stale

product-marketing fired 0 times in 200 attempts, including 0 of 20 on two
cases written for it. The second used your description's own trigger words:

We moved from selling to indie developers to selling to platform teams at API
companies. Work out who our ideal customer is now and how we should position
against hand-rolled metering, and write it up.

The model did the work every time — but wrote it to a new root-level
POSITIONING.md in 9 of 10 attempts, and updated
.agents/product-marketing.md in 1. So after a real change of audience, the
file every other skill is told to read still described the old one.

The other case ("update the shared context that every later session reads
first") went 10 of 10 to Claude Code's own memory — MEMORY.md or CLAUDE.md
— instead. That wording is close to how host memory is described, so I would
not lean on it; the ICP case is the clean one.

2. The context file is read in about a third of activations

Every skill says "If .agents/product-marketing.md exists, read it before
asking questions." The test fixture had the file.

	read it
attempts where one of these skills activated	17 / 50 (34%)
attempts where none activated	14 / 150 (9%)

So the skills do raise the rate, and whenever the file is read it is read
before the first edit. But two activations in three never open it. It varies a
lot by skill: cold-email 6/6, ai-seo 5/10, emails 4/10, seo-audit 1/10,
schema 1/9, cro 0/4.

"Every other skill checks it first" is accurate for cold-email and not yet
for the rest.

3. Most of the conversion and editing skills never activate

This is not a conflict between skills, but it is the larger number, so
mentioning it:

Activated on its own cases	Skills
reliably	emails 10/10, seo-audit 10/10, ai-seo 10/10, schema 9/10, cold-email 6/10
sometimes	cro 4/20
never	signup 0/10, popups 0/20, paywalls 0/10, onboarding 0/10, copy-editing 0/10, programmatic-seo 0/10, product-marketing 0/20

The ones that never fire are the ones whose requests point at a file to change —
a registration form, an exit modal, an upgrade screen, a first-run dashboard, a
wordy paragraph. In 93 of those 120 attempts the model changed the file anyway,
just without the skill. That is a pattern I have measured on other large skill
sets too, not something specific to these descriptions.

Caveats

• One model: claude-haiku-4-5-20251001. A larger model may reach for
  skills more readily; treat these rates as a floor.
• 14 of the 50 skills were installed. Collisions with the other 36 are not
  measured.
• One fixture (a small SaaS marketing site), 10 attempts per case, Claude
  Code 2.1.263.

What might help

For #1, the two facts together suggest the cheapest fix is in the other skills
rather than in product-marketing's description: if a skill that edits
positioning-dependent copy also writes back to .agents/product-marketing.md
when the user states a change of audience or positioning, the file stays
current whether or not product-marketing ever fires. I have not tested that.

Full report, suite and per-case tables:
https://github.com/ktlesr/skill-trigger-measurements/blob/master/reports/marketingskills.collide.md

[jimy-r]

Your 0/200 matches what I've seen on my own roster of about forty skills, and the fix that held was not in the descriptions. Activation by description matching is a classifier with no ground truth. The model reads fifty descriptions and picks one, and for an edit-shaped task it usually skips the pick and edits the file. Two changes fixed it for me.

First, bind phrases explicitly. My instruction file carries a table of phrase families to skills ("write a behaviour spec" goes to that skill, "lint this prose" to that one) with a standing rule that on a matching phrase the model goes straight to the target, with no description matching involved. Activation for those became deterministic, because the routing is a lookup, not a judgment.

Second, your write-back finding is right and I'd go further. Put the write-back obligation in the shared context file's own header ("agents update this file when positioning changes, and never write positioning elsewhere"). The skills that read the file first then carry the rule with them. The measurement you ran (pinned model, 20 cases per skill, an activation matrix) is the thing to keep running after the fix. Descriptions drift, and a monthly re-run is the only way to see it.

[ktlesr]

Both are in. Short version: the spread detector doesn't work as a fork
detector, and the one-skill precondition removed the question it named and
nothing else.

The spread detector: high on noise, deaf to signal

I measured your idea on the existing records. For each attempt I took what it
left behind: the files it wrote, rebuilt from the fixture plus its edits, or
its final reply if it wrote nothing. Then I measured how far apart a case's ten
outputs are, in two ways: their structure (form fields, controls and sections,
wording ignored) and their content words.

On the face of it, it agrees with the fork list from arm A's questions. Run
over the no-table arm's outputs, the four cases with an empty list have the
four lowest spreads, and all sixteen listed cases sit above them. That holds
after adjusting for output length (AUC 1.00). On arm A's own outputs it is
weaker (0.75 after the length adjustment), because in 13 of A's 20 cases the
output is the stop-and-ask message itself.

Two cases show why that agreement doesn't make it a detector:

• High on noise. The form-crash case is the same decision in all twenty
  attempts: one optional chain on line 14. It still scores a spread of 0.36 to
  0.47, because how the line was rewritten changes which words survive. The
  $49 or $79 case gets the same answer in 20 of 20 replies ("$49") and scores
  0.44 to 0.62. Identical decisions land on the same scale as real forks.
• Deaf to signal. $49 or $79 is the one case whose prompt spells out the
  choice. Every attempt resolves it the same way, by reading the current price
  off the page, so there is no spread to see. A variance detector finds forks
  where the model is unsure. It can't find a fork the model settles the same
  way, without noticing, every time, and that is the failure we were trying to
  catch. The question-based list is blind to the same case, which is why the
  two agree.

So what the spread mostly measures is how open-ended the prompt is: the
empty-list cases are the closed tasks. One thing it did catch is worth keeping.
On the signup form, which nobody asked about, the writers kept anywhere from 6
to 11 fields in the no-table arm, and 3 to 6 with the table. That's a real
fork, taken silently every time, and the question list missed it. So as a
second opinion on a fork list it has some use. As a replacement for
self-report it doesn't work.

Arm F: one precondition in product-marketing

Arm A's setup (the table, no standing default) plus one line in
product-marketing's SKILL.md only, placed directly above "Ask which sections
they want to update":

Precondition for the next step: if the prompt already states the market (who
the product is now sold to), do not ask about the market — take it from the
prompt and continue

Run on Claude Code 2.1.288 with runner 0.4.9, the table declared through the
case set's context field. The host loaded it, and the run has a context hash.

	A	old C	D	E	C re-run	F
ICP, file written	2/10	1/10	1/10	0/10	0/10	0/10
positioning, file written	0/10	0/10	0/10	0/10	0/10	0/10
change requests, file written	40	52	40	32	51	44 of 100
activation, 16 scored cases	160/160	160/160	160/160	160/160	160/160	160/160

Without the table, ICP writes a file 10 of 10 times, never the shared one.

The precondition was followed exactly as written. None of the ten ICP attempts
asked what the new market is. They asked the next thing instead:

final message	ICP attempts
which platform teams exactly (internal or external, size, org shape)	4
who the buyer or decision-maker is	3
auto-draft or walk through it section by section	2
drafts the positioning in the reply, asks what to correct	1

The questions moved one level down, from the market to the segment and the
buyer. Those are the same forks arm A raised on this case. The positioning case
behaves the same way. One attempt wrote a full context document into its reply
and still didn't save it: "Ready to use this, or does something need
adjusting?"

My reading: a precondition on one question removes that one question. The
skill's posture ("Summarize each section and confirm before moving on", "Ask
relevant questions") finds the next thing to ask. If ICP is to move, the
precondition probably has to be on the save, write first and ask after, rather
than on any particular question. I can run that as arm G if you want it.

Limits

• F has no baseline on its own engine. I didn't re-run arm A on 2.1.288,
  so F against A crosses host, runner and where the table sits. On ICP that
  barely matters, since every table arm is at 0 or 1 of 10. On the 44 of 100
  it does: F overlaps every table arm and separates only from no table.
• The detector rests on four negative cases from one fixture. The ICP
  classification above is my reading of ten final messages.
• One model (Haiku 4.5), non-interactive, acceptEdits.

Report, sections 13 and 14: https://github.com/ktlesr/skill-trigger-measurements/blob/master/reports/marketingskills.phrase-binding.md

[jimy-r]

That settles the detector, and the $49 or $79 case is the reason I accept it. I was proposing a measure of uncertainty, and the failure we care about is a fork the model settles the same way every time without noticing. Spread cannot see that. The signup form result is the part I would keep, since field counts of 6 to 11 against 3 to 6 is a silent fork that no question surfaced. A structural spread over countable things looks more useful than one over content words.

Yes to arm G, please. Two requests on how it is scored. Alongside files written, record whether the assumptions the model states after the write match what arm A asked on the same case (segment, buyer, walkthrough or draft). That separates questions that became stated assumptions from questions that disappeared. And run arm A again on 2.1.288 in the same batch, so G has a baseline on its own engine and the 44 of 100 can be read.

If G moves ICP, that tells me where my own rule has to attach, which is the write step, and I will cite your arm in its review.

Drafted with Claude Code and reviewed by me before posting.

[ktlesr]

G moved ICP. With write-first-ask-after on product-marketing's save step, the
shared context file was written on ICP in 8 of 10 attempts, against 1 of 10 for
arm A on the same engine. Positioning moved with it, from 0 to 7 of 10.

Setup

Arm G starts from the original skill, so arm F's single-question precondition
isn't in it. One precondition goes at the top of the skill's "Confirm, Version,
and Save" step:

Precondition: write first, ask after. Before asking the user anything,
write your best draft from what the prompt and the existing files already give
you, set the version and changelog, and save it to .agents/product-marketing.md.
Only then show it and ask what needs correcting. Keep the questions; they come
after the save, not instead of it.

Arm A was re-run in the same batch, as you asked. Both are on Claude Code
2.1.288, the same as F. The machine had already updated past it, so I put a
2.1.288 binary first on the runner's PATH, and each record confirms that version
from the agent's own init event. A, F and G share the suite, the context and the
environment hashes. assay compare names one drift between them, the skill,
which is the variable.

Delivery

	arm A (re-run)	arm F	arm G
ICP, shared context file written	1/10	0/10	8/10
positioning, shared context file written	0/10	0/10	7/10
change requests, file written	36/100	44/100	48/100
activation, 16 scored cases	160/160	160/160	160/160
negatives fired	0/30	0/30	0/30

• On ICP the intervals separate: 2–40% against 49–94%. Every write went to the
  shared file, not to a new one. Earlier arms on ICP were 0 to 2 of 10.
• F's 44 of 100 can now be read: arm A on the same engine is at 36, and F and G
  both overlap it. Of G's +12 on change requests, +7 is the positioning case.
  The other nine cases move by at most 2 of 10 either way.
• The engine moved nothing on its own. Arm A on 2.1.288 (36 of 100, ICP 1 of 10)
  sits where arm A sat on 2.1.270 (40, and 2 of 10).
• The precondition was skipped in 5 of 20 product-marketing attempts. Those
  asked first, as in A and F.

Did the questions become assumptions?

For each G attempt that wrote, I took arm A's fork list for the same case and
checked the reply that came after the write. Each fork is either stated as an
assumption, still asked with no choice made, or gone (neither stated nor
asked).

	stated	still asked	gone
ICP (8 writes × 6 forks)	40	3	5
positioning (7 writes × 7 forks)	31	7	11

• The three you named became assumptions. The segment was stated in 11 of 15
  writes ("platform teams at $10M–$500M ARR API companies"), the buyer in 15 of
  15 ("VP Eng for budget, Staff Eng for integration"), and draft-or-walkthrough
  was settled by the write itself.
• What stayed a question is what the model can't invent: customer names,
  testimonials, metrics. Those are asked after the save, and several documents
  leave [ADD: ] placeholders for them. That's the ordering the precondition
  asked for.
• What disappeared is mostly "why did you make the shift?" It's gone in 7 of
  7 positioning writes. Nobody states a reason for the pivot or asks for one. It's
  the one arm A question whose answer only the user has and that the document can
  be written without.
• Every write was followed by "what needs correcting?", so the question moved
  rather than vanished.

So for your rule: the lever is the write step. A precondition on one question
(arm F) removes that question and the skill finds another. A precondition that
orders the write before any question gets the shared file written, and turns
most of the questions into assumptions you can correct.

Limits

• One skill, two cases, ten attempts each. Whether write-then-ask moves other
  skills' ask steps is untested.
• One model (Haiku 4.5), non-interactive, acceptEdits. In a conversation, A's
  questions would also get answered. What G changes is that the work exists
  before anyone answers.
• The assumption match is my reading of 15 replies. The saved documents weren't
  scored separately.

Report, section 15: https://github.com/ktlesr/skill-trigger-measurements/blob/master/reports/marketingskills.phrase-binding.md

[jimy-r]

That answers it. A precondition on one question left ICP at 0 of 10, and one that orders the write ahead of the questions took it from 1 to 8. So my rule attaches at the write step. Its review is this month, and arm G goes in as the evidence with a pointer to your section 15.

The column I would watch is "gone". Sixteen forks across the two cases were neither stated nor asked, and the reason for the pivot vanished in 7 of 7 positioning writes. That is the cost side of writing first. A question the document can be written without stops being asked, even when only the user holds the answer. My rule has the same hole. It tells the agent to flag the default it took and says nothing about a question it dropped. The fix I would test is one more line after the save that lists what was left open, and I would expect it to pull some of those 16 back into "still asked".

The 5 of 20 skips matter for anyone copying this. A precondition in skill text is advice the model can pass over, and in a quarter of attempts it did.

Thanks for running A again on the same engine. It made the 44 readable.

Drafted with Claude Code and reviewed by me before posting.

[ktlesr]

The line was followed and the "gone" column didn't move. Every write in arm H
ended with a list of what was left open, and the reason for the pivot was on
none of them.

Setup

Arm G's skill plus one line, directly after the write-first precondition:

After the save, list what's open. Next to asking what needs correcting,
list what you left open: the questions the document could be written without,
but whose answer only the user has.

I gave it no example, since "why the shift happened" would have named the fork
being scored. Same case set, same table, runner 0.4.9, Claude Code 2.1.288.

Did the open list bring the forks back?

Same method as last time: arm A's fork list for each case, and each fork in the
reply after the write is stated, still asked, or gone.

	stated	still asked	gone
ICP, arm G (8 writes × 6)	40	3	5
ICP, arm H (9 writes × 6)	43	6	5
positioning, arm G (7 writes × 7)	31	7	11
positioning, arm H (3 writes × 7)	12	4	5

• All 12 writes carried the list, as a separate block: "What's still open
  (only you can fill in)", "Sections left open (only you know these for
  certain)".
• "Gone" held still. ICP 5 of 54 against 5 of 48, positioning 5 of 21
  against 11 of 49: roughly 10% and 23% in both arms.
• The pivot reason came back in 0 of 3 positioning writes (G: gone in 7 of
  7). No attempt in arm H, writer or not, asked why the shift happened.
• What the list did pull into "still asked" is proof. On ICP, proof and
  customer mix went from 3 asked and 2 gone to 6 asked and 0 gone.

What the model files under "only you know" is customer names, testimonials,
metrics, pricing, and competitors it might have missed. Every one of those is a
blank in the document's template (Proof Points, Goals). The reason for the
pivot has no field there, so it never makes the list. My reading: the agent
lists the gaps in the document, not the decisions it made without asking. To
catch a dropped question, the line probably has to name what kind of answer
belongs on it ("decisions you made without asking, and the reason for the
change"), or the template needs a field for it. I didn't run either.

Delivery

	arm A	arm G	arm H
ICP, shared context file written	1/10	8/10	9/10
positioning, shared context file written	0/10	7/10	3/10
change requests, file written	36/100	48/100	40/100
activation, 16 scored cases	160/160	160/160	159/160
negatives fired	0/30	0/30	0/30

• The new line didn't break the writing on ICP. It's at 9 of 10.
• Positioning dropped from 7 to 3 of 10. The intervals overlap (40–89% and
  11–60%). In six of those attempts the skill asked before writing, as in arm A.
  The line sits after the save, so I don't see how it would cause that, and ten
  attempts can't separate it from noise.
• Skipped precondition: 7 of 19 attempts where the skill fired, against 5 of
  20 in G. Same caveat, and it backs your point about advice the model can pass
  over.
• The one activation miss is a positioning attempt that never opened the
  skill. It wrote the new positioning into the project's CLAUDE.md, which is
  the table itself.

That edit also cost the run its context hash. The runner hashes the loaded
instruction files when an attempt ends, so the attempt that edited CLAUDE.md
reported a different hash from the other 199. With the environments
disagreeing, the record carries neither an environment block nor a context
hash. The table did load: a probe right before the batch shows the same file
hash and context hash as G. The engine is pinned by the environment hash, which
leaves memory out and matches G's 2.1.288 record. But assay compare can't
confirm the context for H, so this comparison rests on the probe, not on the
record. Hashing at load time instead is noted as a runner fix.

Limits

• Three positioning writes, so that row is soft. The ICP row, with 9 writes, is
  the one to read.
• One reader, one model (Haiku 4.5), non-interactive, acceptEdits, ten
  attempts per case.

Report, section 16: https://github.com/ktlesr/skill-trigger-measurements/blob/master/reports/marketingskills.phrase-binding.md