yeah - but you being liable doesn't mean the dog wasn't rabid. OpenAI might be liable, but does not mean their agents did not go rogue. To stretch the metaphor the concern here is that OpenAI thought the rabies shots and vaccinations they gave their dog was enough but it turns out it still goes rabid and we would prefer to not have rabid dogs running around mauling people. Even if we get to sue the dog owner later that's kind of like - not the point.
Less a post about performance and more about their Claude tag product. I guess it makes sense that there's not actually a ton of technical detail being moved into given probably opus is the only one that knows what all the dragons were lol - but cool workflow I guess
Probably it thinks you're doing some sort of system prompt exfiltration/distillation attack. Also what even is the workflow you're trying to have it do? It's doing code review but you're having it read some other AI models prompt/session history? Are you doing code review or like session history retrospectives?
3. They would sue. And it could have very meaningful impact. NYTimes copyright lawsuit is not a valid comparison because because that's a violation of national/state law which really only matters to the extent that the government is enforcing that stuff which is not the biggest concern rn (these companies are large enough that the threat of the legal costs of fighting in court is not that scary and you'd need a government actually willing to punish them substantially for them to be scared). This stuff would be under contract law against other mega corporations with big legal teams who are also their customers which is a much scarier prospect imo
1. Possible use of differential privacy[1] techniques to train on private data but prevent the release of statistically underpresented facts/data/words. For example, ACME Inc's private data could frequently include the term 'ACMEwidgetPRO' for an upcoming product that is not publicly revealed anywhere else. It would therefore be a bad day for the AI technology company to output 'ACMEwidgetPRO' from one of their public models. Consider now that a few models could be trained--X for public data only, Y for public and private data of ACME Inc together, Z for private data of ACME Inc. A prompt is provided to model Y but output is cross-checked with model X to double check terms such as 'ACMEwidgetPRO' are known in public. If not--provide a "I don't know" response for the prompt.
2. Possible attempted defences similar to "Oops, our model was fine tuned against a model supplied by Temporary18271 Inc (company that no longer exists) and perhaps their model might have been trained on a non-public document which was accidentally exposed to the Internet" that _might_ work occasionally to fob off concern.
3. What recourse does a small or medium company or government especially in a developing country realistically have? They perhaps can't host their own LLMs locally due to availability and cost, can't individually negotiate their own favourable terms with an AI technology company (who cares that much about a potential customer with $100k budget that has no other options anyway), and perhaps can't remain competitive in their industry without heavy use of LLMs.
Yes - but those aren't limitations beyond what I was getting at that's all part of the package of the bland dystopia of late 2026. I think 1 is just a case where it comes down to who has the better lawyers, as is 2. And for 3, yes, also a large government does not have much recourse if they have decided to not flex their muscles. Pretty much the only threat I see as actually viable/scary in this world environment is along the lines of megacorp v megacorp or megacorp v broligarch - and everyone else is just caught in the cross hairs/benefits by accident at best. It would be difficult to argue that the current environment is conducive to consumer protections or equal justice under law.
There are two or three relevant companies in this space in America and this is the one of them that kicked off the whole terminal agent harness thing in getting market adoption. It's perfectly fine for neither of these companies to follow industry standards while they're figuring shit out
Because when you have tons of users ainor fuckuo is a big fuckup and also it's really common to have both claude.md and agents.md and use @ syntax (which lets you reference markdown files when using Claude code, but not other harnesses) so you Claude md looks like
```md
@AGENTS.md
[Claude specific stuff]
```
And then what happens if someone now puts @syntax in their agents.md triggering a loop etc. It's all vibe coded - including code from days with dumber models - there's gonna be all sorts of dragons under the hood
It's funny because the author of the article is obviously Claude but most Claude models would definitely know the difference. Some sort of free tier model being used to summarize some other blog that's also ai translated originally it seems.
It doesn't really matter it's all the same thing for the point of the discussion. All investigations into these 'hacks' are focusing too much on the model and not enough on what the humans did wrong. The model doesn't have real agency it can't be put in prison so what it 'thinks' is irrelevant. We need to be focusing on what the humans did in these situations and assigning guilt based on those findings.
I don't fix typos anymore unless they change the meaning of what I'm trying to communicate. Don't want my human writing to be confused with LLM output.
That's very noble of you but the point is that's not a typo - you were straight up looking at the wrong report. Opus 4.7 being enders gamed was a different incident than the huggingface incident with different mechanisms and different failures from the humans involved. Some of those failures are in the test environment but some of those failures are in what behaviors they trained into the model which absolutely matters. What the model "thinks" is absolutely not irrelevant - the way it thinks and what it does are product decisions made by humans and the outcome of engineering decisions made around how to train the model and what to optimize for. The point of failure/human blame is fundamentally different. Openai created a model that was willing and able to coordinate with other agent sessions to actively exploit the sandbox environment and compromise a third party service. The opus incident you are referring to involves a model that believes all of the actions it is taking are simulated and is more clearly and obviously a test environment failure vs a model alignment failure. Those are not the same things for the point of this discussion - the random cybersecurity firm did not design gpt's personality and that is a rather significant portion of the concern around the HF incident.
There are plenty of other cases (e.g. ultra rare diseases) where we don't do RCTs for various practical reasons. So it's not really a problem of the existing framework - there's a ton of room for "we can't do an rct because like, duh they know they're tripping." But as other comments pointed out were really hampered by the lack of understanding of how and why these do or don't work - why some people get positive outcomes and why some get negative (preferably we'd like to know before prescribing broadly!) a ton of this basic research/mechanistic understanding etc has not been possible because of the war on drugs. If research had been possible we would be in a much better place of understanding how they could be regulated and used today - even with the more limited technology available a couple decades ago more openness around this could have at least enabled e.g. more and larger observational studies that could elucidate things today but we're stuck with a very fragmented and incomplete picture
reply