Infosys chair bets companies will develop their own AI models

An example of a fine-tuned LLM getting good results versus a human:

LLMs trained on the vast scientific literature could potentially integrate noisy yet interrelated findings to forecast novel results better than human experts. Here, to evaluate this possibility, we created BrainBench, a forward-looking benchmark for predicting neuroscience results. We find that LLMs surpass experts in predicting experimental outcomes. BrainGPT, an LLM we tuned on the neuroscience literature, performed better yet. Like human experts, when LLMs indicated high confidence in their predictions, their responses were more likely to be correct, which presages a future where LLMs assist humans in making discoveries. Our approach is not neuroscience specific and is transferable to other knowledge-intensive endeavours.

They used LoRA to finetune Mistral-7B-v0.1 on neuroscience literature.

Researchers are sharing encouraging early reports about o1-preview as an aid for tackling complex scientific challenges.

Researchers at the national lab have also been surprised by o1-preview’s ability to recognize when it doesn’t have all the necessary information to answer a question and make reasonable assumptions for variables it might be missing, the person said.

The Lawrence Livermore example is similar to the positive reaction Australian-American mathematician Terence Tao shared after the initial release of o1-preview and o1-mini. Tao used the models to solve math problems and write proofs—something that a typical ChatGPT user probably wouldn’t do.

“It may only take one or two further iterations of improved capability” until such a reasoning model becomes a “competent graduate student…at which point I could see this tool being of significant use in research level tasks,” he said.

This dilemma mirrors one potentially faced by junior lawyers: if AI handles graduate-level research tasks, how will the next generation of researchers develop their skills?