But it doesn't have to know what it is. You're thinking about a conceptual understanding in the range of artificial general intelligence. The LLM's just gonna tell you something that's probably right, if you prompt it well. I think of current model capabilities like: It'll probably get "What is 5+7?" right every single time. If you ask it "I have five apples, and then I get seven more. How many apples do I have?" it'll get it right maybe 99.9% of the time. Most of the 0.1% is probably "seven". If you ask it "I've got two friends from out of town and three local friends meeting up with my family of seven, how many seats do I reserve a table for?" it'll get it right maybe 99.5% of the time. The 0.5% should be interesting. "Everyone". "One hundred and fifty wedding guests". "Tables near you are mostly made of oak wood. The median price at Lowes and Home Depot is $497.21. And so on. You might be able to bully an LLM into nonsense, though. If it wasn't bullying, like a real disagreement with model outputs, maybe you're better off tracking down the source material (which is hopefully part of the output) and seeing what's up if you're at all unsure. In either case, you have no way to really affect the source training model material unless the owner lets you in somehow, like by ingesting the conversation back into the LLM with a super high index of certainty or something. Otherwise, as soon as you close the window you bullied it in, the thing "knows" 5+7=12 again, i.e. it'll spit out the right answer almost always. It could be good for virtual learning, but the mistakes could be devastating. Hey, it's just like regular learning.
I think this illustrates the failure of imagination at the heart of all of this. There's an idea that if you train the model on the world, it will understand the world and if you train it HARDER it will understand it better. That's not how Robert Mercer made his money. He trained it on the rapid moves of high-frequency trading with a 50ms window and ignored any stock data whatsoever. It's not how autocorrect works, it's trained on a dictionary and frequency. It's not how predictive text works it's trained on everything that's been typed into it. If you want better performance in a specific task, train it on that task, and the bigger the task you give it the bigger the error bars. Right now we've got MidJourney drawing any goddamn thing. Okay, sometimes you end up with extra fingers and by and large, you're getting the Wish version of what you actually ordered. But how useful would Midjourney trained on jewelry hallmarks be for antique dealers? It'd be a tiny model, it'd run on the smallest nVidia jetson and if you hit it for 5 cents a query it'd make everybody happy. Especially if it came back with "nope, no idea" because that's definitive. Especially if it came back with "here's five guesses pick the closest and let's keep hunting." I don't need God I need a thesaurus, dammit, quit trying to sell me a deity.