Instead of a "one size fits all" approach, different treatments were offered depending on the severity of the condition.
Writing in the Lancet, researchers said the findings would have "important implications" for back pain patients.
Other doctors said the findings were "very promising".
Estimates for the UK suggest back pain is very common with between 6% and 9% of adults visiting their GP with the condition each year. Up to 80% say they still have pain a year later.
The report's authors argued that current care, such as advice, painkillers or referral to a physiotherapist, was based on "intuition" which was "inefficient and inconsistent".
In the study, 283 patients were treated normally and 568 were assessed as having low, medium or high risk of long-term and disabling back pain.
Low-risk patients were given advice by a physiotherapist. Medium-risk patients received intensive physiotherapy. High-risk patients had physiotherapy and help to overcome psychological barriers to recovery, which can occur in chronic conditions.
After 12 months, the targeted groups showed significantly more improvement than the traditional treatment group.
The study said the benefit was achieved at lower cost - saving more than £30 per patient on average.
Lead researcher Prof Elaine Hay, of Keele University, said: "The problem for GPs and other health professionals is spotting who, amongst the patients they see with back pain, is likely to get better with simple advice and reassurance, and who might benefit from further treatment from a physiotherapist."
She said 50 centres in the UK had already adopted the new approach: "The data shows that it is having a substantial beneficial impact where it is being implemented."
The study was funded by Arthritis Research UK. Its medical director Prof Alan Silman said there had been many studies into the effectiveness of exercise and physiological treatments, but: "This research is novel because it shows how to take the best of these two approaches, and make a real difference to large numbers of patients."
Dr Bart Koes, of Erasmus University Medical Centre in The Netherlands, said in an accompanying article that "further improvements are desirable" but "these results are indeed very promising".
He added: "The economic assessment showed that the new approach was cost effective so there is no financial reason not to implement."