Abertawe Bro Morgannwg University Health Board (ABMU) said patients could face significant difficulties seeing their doctor as the population grows.
The proposals would place GP, dental, research, fitness and library facilities under one roof.
ABMU said move was the best way to “promote good health”.
Sian Harrop-Griffiths, ABMU’s director of strategy said the proposals were still in the early stages and nothing was set in stone.
“The NHS is facing rising costs, increasing demand and an ageing population, by working across all sectors we aim to tackle these issues by putting prevention and improved health and wellbeing in a joined up way,” she said.
Discussions are taking place to see if the wellness centre could be part of theSwansea city centre regeneration plans.
A report being considered by Swansea Public Services Board – a partnership of local agencies including Swansea Council and ABMU – says the hub would help ease pressures at GP surgeries, many of which are out-of-date and “unsuitable” for dealing with modern health care provision.
It warns that problems recruiting GPs teamed with increasing demand threaten the sustainability of primary health care in the city.
Earlier this month the British Medical Association (BMA) warned primary care was in “crisis” in Wales, saying more family doctors could quit unless surgeries get more support.
The proposed centre would provide GP services for 35,000 patients as well as housing dental, sexual health, podiatry, speech and language and mental health services for around 150,000 people.
A GP academy, library, fitness facility, art therapy and computer courses could also be based in the building.
Proposals for the 7,000 square-metre centre are being looked at by Swansea Public Services Board as part of the wider regeneration plans.