The Resolution Foundation said rising numbers of people in economic inactivity must be addressed as Treasurer Jeremy Hunt aimed to get people back to work with his Spring Budget this week.
Census figures from the Office for National Statistics show 40,520 residents in Waverley were economically inactive between March 15 and 22 2021.
An economically inactive person is aged 16 and over and did not have a job in that time period, could not start work in the next two weeks, or had not looked for work in the month before.
It meant 39.1% of over-16s in the area were economically inactive.
Across the two countries, 19.1 million over-16s (39.4%) were economically inactive.
The highest proportions tended to be in coastal areas, such as East Lindsey in Lincolnshire, North Norfolk and Tendring in Essex, all of which had more economically inactive than active residents.
Meanwhile, the six most active areas were all in London, and every area in the top 20 was in London or the South East bar commuter town Watford.
Louise Murphy, economist at the Resolution Foundation, said: "Levels of inactivity vary significantly across the UK, and tend to be highest in places where the population is older, and where long-term illness is widespread, such as deprived, ex-industrial areas.
"Economic inactivity is also concentred in low-income households. One-in-three adults in the poorest tenth of households have a disability, compared to fewer than one-in-ten adults among the richest families.
"The right approach to boosting workforce participation should therefore focus both on tackling place-based inequalities, as well as raising living standards for the poorest families."
Amanda Walters, director of the Safe Sick Pay campaign, highlighted rising numbers of people not working due to long-term sickness or disability.
"Help from the Government to support these people back into work will be very welcome, but people fear what happens if they get ill again," she added.
"The 'back to work' Budget had the right focus, but it was an own goal from the Chancellor not to make sick pay available for everyone from the first day of illness."
The census figures show 2,405 Waverley residents were not working or looking for work due to long-term sickness or disability in March 2021.
A further 25,620 had retired.
It meant long-term sickness or disability accounted for 5.9% of the economically inactive people in the area, while 63.2% were retired.
Chancellor Jeremy Hunt said he will deliver growth with his 'back to work' Budget by "removing obstacles that stop businesses investing, tackling labour shortages that stop them recruiting, and by breaking down barriers that stop people working".
Mr Hunt added that he will encourage the long-term sick and disabled into work, including announcing the Universal Support programme – a voluntary employment scheme for disabled people where the Government will spend up to £4,000 per person to help them find appropriate jobs.
Schemes to encourage young carers, older people nearing retirement and parents – including 30 hours of free weekly childcare for all under-5s – to work were also introduced.
"Our plan is working – inflation falling, debt down and a growing economy," Mr Hunt added. "Britain is on a lasting path to growth with a revolution in childcare support, the biggest ever employment package and the best investment incentives in Europe."