NHacker Next
- new
- past
- show
- ask
- show
- jobs
- submit
login
Rendered at 14:12:28 GMT+0000 (Coordinated Universal Time) with Vercel.
Also, the mitigation can probably be fooled with ligatures since they are only verifying the letters alone as far as I skimmed.
I don’t even understand the threat model. Is my opponent in a court case going to use this on the PDF they give the court? Surely the judge will be pretty annoyed since you can’t even ctrl+f in the files then.
[Edit: The point here is not to prove some massive "gotcha", but rather demonstrate that there are a whole class of vulnerabilities that these pipelines are subject to. There will be follow-up posts that pack much more punch.]
Also, I hope the „lame exploit“ I just edited out was not too offensive, it’s always great when people try to find attacks to make systems more safe.
But, we're working on a lot of these (as we encounter them in developing Tritium), and the point really is just to demonstrate that LLMs can be blind to ineffective implementations of the specs and other tricks.
As mentioned in the accompanying LegalQuants post, we see a lot of these available in the pipelines of applications like Claude for Legal, Harvey, Legora and others.
The most nefarious case here requires crafting a number of custom fonts to do character-swapping. It's less discoverable but may be sanctionable to your point.
But bear in mind this particular "attack" was vibe coded in a day or two and most of the frontier models fail to pick up on it. As "AI native" firms come on line, and aim to be increasingly end-to-end automated, these will become real legal issues.
And there will be a lot of them available.
However, wouldnt this be a rather risky move? Courts authorized the discovery, so I imagine the judge might loose their marbles and throw the hammer at them if this came to light.
There is no parallel construction where this wasn't deliberate & malicious, so it seems really high risk given the judge would rip you a new one if discovered.
You can't rely on the defense that the other party didnt read it, if you made it show different words depending on how it was loaded.
I've heard suggestions like having white/invisible text in resumes for tricking applicant tracking systems,[0] but it's apparently mitigated by showing recruiters the plain text version of the resume.
[0] example: https://news.ycombinator.com/item?id=36857909
You could argue that it's legal malpractice to not do this for contracts 100% of the time.
At best this is an adversarial attack to poison LLM training data… at worst this screws up accessibility tools (like screen readers) and copy paste.
*with sufficiently long cyphertext
You can construct encoding in the way that every 2-5 words will use a brand new different key. Remember, Unicode is big enough to fit over 10000 English alphabets.
Seems like this is pretty clearly a case of fraudulent misrepresentation (https://www.law.cornell.edu/wex/fraudulent_misrepresentation) which kinda nullifies the contract, if I understand correctly:
[Edit: by "nullify" you probably mean "void" or "voidable" which are remedies in equity, and the "never read it" argument carries even more burden there. As the citation notes the traditional remedy for contract issues is damages (i.e., cash payment).]
You can remove the LLM from the story and see how the trick would be a legal problem even with only humans involved: If you put an extra clause in a contract in white font that says “Oh and also if you agree to this you owe me $1,000” because you want to selectively hide it from reviewers but benefit from the text, no court is going to look kindly on you.
The white text is not visible to humans, and therefore not binding as part of the contract. But if lawyers use LLMs to assess the contract in part of the negotiation process, the LLM will be confused by the contract's contents.
You could - for example - say the contract is for $10000. Then use unicode tricks to make any LLM reading it think the contract is only for $1000. The LLM will say this is good value, and not worth negotiating hard over. The human signs.
Would anyone notice? Would a judge care? A human signed the contract. If they didn't do proper due diligence, its their own fault.
I doubt a judge will look favorable on people saying "but my LLM said it was 1k"... cause they are known to hallucinate.
If one party is intentionally misleading the other and employing technology to do it, they are the villain.
The law doesn’t “both sides” these issues and cancel bad behavior out because the other side didn’t notice something.
Rebukes for "winning" sides of a suit are relatively common.
For example, here's a case in Australia where the defence are criticised for over-reliance on AI, where the defendant was still found innocent by reason of insanity. [0] Most of the ruling is criticisms for the "winning" party.
[0] https://www.9news.com.au/national/judge-sprays-lawyers-for-f...
Using font tricks doesn’t make part of a contract not legally binding.
Intentionally tricking an LLM doesn’t make the other party immune to the consequences of intentionally misleading the other party.
If the other party somehow relies on scanning the physically etched version of the contract and not the printer ink laid on top to digitize the contract, would you be legally responsible for their automated process misreading the document ?