Model Fatigue
Seen the other day on LinkedIn: To get even faster news about which AI model is the best, don't wait for press releases or reading about it on the news. Instead, get real-time insights by checking prediction markets. Presumably, insiders will give it away by placing bets, causing spikes in market activity on a bet such as "Who'll have the best AI model by the end of July?"
Leaving aside the myriad ethical issues with prediction markets (John Oliver did a great segment on them in his show, Last Week Tonight), and leaving aside that price spikes in prediction markets could also just be random noise, what plausible scenario exists where you need to know, down to the minute, if one AI lab versus the other will have "the best" model out?
Know which model is going to be the best one by the end of this month before anyone else does
???
Profit
Can someone please fill in the blank for me? I don't see it.
These days, the model is just one small piece of a larger puzzle of plugins, skills, harnesses and all the other carefully built and curated things that make the overall system run well. Time spent chasing leaderboards is time not spent improving the system you have right in front of you.
"But what if I miss something really big?"
That's the good part. You don't. If it's big enough, you'll hear about it. The lag is a feature, because it filters out the noise and the fads.
