It stands to reason that if you have access to an LLM’s training data, you can influence what’s coming out the other end of the inscrutable AI’s network. The obvious guess is that…
Yeah, after reading a bit into it. It seems like most of the work is up front, pre filtering and classifying before it hits the model, to your point the model training part is expensive…
I think broadly though, the idea that they are just including the kitchen sink into the models without any consideration of source quality isn’t true
I’m sure that’s true, but it is also noteworthy that any and every consideration that goes into the initial inclusion of the data before it is fed into the model introduces intended and unintended consequences on the training.
Furthermore, the proliferation of the LLMs themselves is putting negative pressure on survival of the places where all the good data is sourced from in the first place. When traffic to a place like stackoverflow is way down because everyone’s reading LLM answers (that the LLM training dataset got from stack overflow), there are less good conversations on stackoverflow to read. Some of these sources of training data may even be caused to cease to exist entirely.
Yeah, after reading a bit into it. It seems like most of the work is up front, pre filtering and classifying before it hits the model, to your point the model training part is expensive…
I think broadly though, the idea that they are just including the kitchen sink into the models without any consideration of source quality isn’t true
I’m sure that’s true, but it is also noteworthy that any and every consideration that goes into the initial inclusion of the data before it is fed into the model introduces intended and unintended consequences on the training.
Furthermore, the proliferation of the LLMs themselves is putting negative pressure on survival of the places where all the good data is sourced from in the first place. When traffic to a place like stackoverflow is way down because everyone’s reading LLM answers (that the LLM training dataset got from stack overflow), there are less good conversations on stackoverflow to read. Some of these sources of training data may even be caused to cease to exist entirely.