It stands to reason that if you have access to an LLM’s training data, you can influence what’s coming out the other end of the inscrutable AI’s network. The obvious guess is that…
I mean, if they didn’t piss in the pool, they’d have a lower chance of encountering piss. Godwin’s law is more benign and incidental. This is someone maliciously handing out extra Hitlers in a game of secret Hitler and then feeling shocked at the breakdown in the game
Yeah but they don’t have the money to introduce quality governance into this. So the brain trust of Reddit it is. Which explains why LLMs have gotten all weirdly socially combative too; like two neckbeards having at it—Google skill vs Google skill—is a rich source of A+++ knowledge and social behaviour.
If I’m creating a corpus for an LLM to consume, I feel like I would probably create some data source quality score and drop anything that makes my model worse.
Then you have to create a framework for evaluating the effect of the addition of each source into “positive” or “negative”. Good luck with that. They can’t even map input objects in the training data to their actual source correctly or consistently.
It’s absolutely possible, but pretty much anything that adds more overhead per each individual input in the training data is going to be too costly for any of them to try and pursue.
O(n) isn’t bad, but when your n is as absurdly big as the training corpuses these things use, that has big effects. And there’s no telling if it would actually only be an O(n) cost.
Yeah, after reading a bit into it. It seems like most of the work is up front, pre filtering and classifying before it hits the model, to your point the model training part is expensive…
I think broadly though, the idea that they are just including the kitchen sink into the models without any consideration of source quality isn’t true
I’m sure that’s true, but it is also noteworthy that any and every consideration that goes into the initial inclusion of the data before it is fed into the model introduces intended and unintended consequences on the training.
Furthermore, the proliferation of the LLMs themselves is putting negative pressure on survival of the places where all the good data is sourced from in the first place. When traffic to a place like stackoverflow is way down because everyone’s reading LLM answers (that the LLM training dataset got from stack overflow), there are less good conversations on stackoverflow to read. Some of these sources of training data may even be caused to cease to exist entirely.
As far as I know that’s generally what is often done, but it’s a surprisingly hard problem to solve ‘completely’ for two reasons:
The more obvious one - how do you define quality? When you’re working with the amount of data LLMs require as input and need to be checked for on output you’re going to have to automate these quality checks, and in one way or another it comes back around to some system having to define and judge against this score.
There’s many different benchmarks out there nowadays, but it’s still virtually impossible to just have ‘a’ quality score for such a complex task.
Perhaps the less obvious one - you generally don’t want to ‘overfit’ your model to whatever quality scoring system you set up. If you get too close to it, your model typically won’t be generally useful anymore, rather just always outputting things which exactly satisfy the scoring principle, nothing else.
If it reaches a theoretical perfect score, it would just end up being a replication of the quality score itself.
like the LLM that was finding cancers and people were initially impressed but then they figured out the LLM had just correlated a DR’s name on the scan to a high likelihood of cancer. Once the complicating data point was removed, the LLM no longer performed impressively. Point #2 is very Goodhart’s law adjacent.
I never knew the name for this law, but it’s basically how SEO ruined traditional search. I think it’s also a big reason that a LOT of software engineers put way too much emphasis on passing unit tests and not nearly enough on examining what they’re actually testing.
i understood that reference to handing out secret hitlers. played that game first during hike called ‘three capes’ in Tasmania. laughed ‘til my cheeks hurt.
So, like with Godwin’s law, the probability of a LLM being poisoned as it harvests enough data to become useful approaches 1.
The problem is the harvesting.
In previous incarnations of this process they used curated data because of hardware limitations.
Now that hardware has improved they found if they throw enough random data into it, these complex patterns emerge.
The complexity also has a lot of people believing it’s some form of emergent intelligence.
Research shows there is no emergent intelligence or they are incredibly brittle such as this one. Not to mention they end up spouting nonsense.
These things will remain toys until they get back to purposeful data inputs. But curation is expensive, harvesting is cheap.
I mean, if they didn’t piss in the pool, they’d have a lower chance of encountering piss. Godwin’s law is more benign and incidental. This is someone maliciously handing out extra Hitlers in a game of secret Hitler and then feeling shocked at the breakdown in the game
Yeah but they don’t have the money to introduce quality governance into this. So the brain trust of Reddit it is. Which explains why LLMs have gotten all weirdly socially combative too; like two neckbeards having at it—Google skill vs Google skill—is a rich source of A+++ knowledge and social behaviour.
If I’m creating a corpus for an LLM to consume, I feel like I would probably create some data source quality score and drop anything that makes my model worse.
Then you have to create a framework for evaluating the effect of the addition of each source into “positive” or “negative”. Good luck with that. They can’t even map input objects in the training data to their actual source correctly or consistently.
It’s absolutely possible, but pretty much anything that adds more overhead per each individual input in the training data is going to be too costly for any of them to try and pursue.
O(n) isn’t bad, but when your n is as absurdly big as the training corpuses these things use, that has big effects. And there’s no telling if it would actually only be an O(n) cost.
Yeah, after reading a bit into it. It seems like most of the work is up front, pre filtering and classifying before it hits the model, to your point the model training part is expensive…
I think broadly though, the idea that they are just including the kitchen sink into the models without any consideration of source quality isn’t true
I’m sure that’s true, but it is also noteworthy that any and every consideration that goes into the initial inclusion of the data before it is fed into the model introduces intended and unintended consequences on the training.
Furthermore, the proliferation of the LLMs themselves is putting negative pressure on survival of the places where all the good data is sourced from in the first place. When traffic to a place like stackoverflow is way down because everyone’s reading LLM answers (that the LLM training dataset got from stack overflow), there are less good conversations on stackoverflow to read. Some of these sources of training data may even be caused to cease to exist entirely.
As far as I know that’s generally what is often done, but it’s a surprisingly hard problem to solve ‘completely’ for two reasons:
The more obvious one - how do you define quality? When you’re working with the amount of data LLMs require as input and need to be checked for on output you’re going to have to automate these quality checks, and in one way or another it comes back around to some system having to define and judge against this score.
There’s many different benchmarks out there nowadays, but it’s still virtually impossible to just have ‘a’ quality score for such a complex task.
Perhaps the less obvious one - you generally don’t want to ‘overfit’ your model to whatever quality scoring system you set up. If you get too close to it, your model typically won’t be generally useful anymore, rather just always outputting things which exactly satisfy the scoring principle, nothing else.
If it reaches a theoretical perfect score, it would just end up being a replication of the quality score itself.
like the LLM that was finding cancers and people were initially impressed but then they figured out the LLM had just correlated a DR’s name on the scan to a high likelihood of cancer. Once the complicating data point was removed, the LLM no longer performed impressively. Point #2 is very Goodhart’s law adjacent.
I never knew the name for this law, but it’s basically how SEO ruined traditional search. I think it’s also a big reason that a LOT of software engineers put way too much emphasis on passing unit tests and not nearly enough on examining what they’re actually testing.
Good points. What’s novel information vs. wrong information? (And subtly wrong is harder to understand than very wrong)
At some point it’s hitting a user who is giving feedback, but I imagine data lineage once it gets to the end user its tricky to understand.
i understood that reference to handing out secret hitlers. played that game first during hike called ‘three capes’ in Tasmania. laughed ‘til my cheeks hurt.
It’s just “mafia/werewolf” by a different name
Hey now, if you hand everyone a “Hitler” card in Secret Hitler, it plays very strangely but in the end everyone wins.
…except the Jews.