Responsibility for rural issues should be transferred away from Defra and given to another government department, says the committee.
The recommendation was made by a House of Lords Committee report called "Countryside at a Crossroads".
The committee has been scrutinising the impact of the Natural Environment and Rural Communities Act 2006 (NERC Act).
See also: Watchdog in final plea over services
A series of cutbacks over the past 12 years has diminished the resources given to departments and bodies which protect the UK’s natural environment and promote the needs of rural communities, it says.
The profound, negative impact this has on England’s biodiversity – and the social and economic welfare of rural areas – must be reversed, it says.
The committee points out that independent bodies created by the NERC Act, such as the Commission for Rural Communities, have been abolished.
Their loss has significantly weakened the government's understanding of rural society, leading to policy changes that fail to account for the negative impacts for people in rural communities.
The committee also found that there has been a consistent failure, over a number of years to prioritise the 'rural affairs' element of the departmental remit of Defra.
The focus of the department has been consumed by its work on agriculture and the environment and this will only intensify as a result of Brexit, it says.
The lives of those who live in the countryside have been neglected as a result, concludes the committee's report.
Responsibility for rural policy should be transferred from Defra to the Ministry for Housing, Communities and Local Government, it recommends.
This ministry already delivers many of the key services that support rural vitality.
Lord Cameron of Dillington, Chairman of the Committee, said: "It is clear that the government are failing to take proper account of the needs of rural communities.
Lack of understanding
"Departmental decisions and policies continue to demonstrate a lack of rural understanding among Whitehall policymakers.
"Each and every government department should be required to think about the ways in which their policies affect rural people, and the government must take action to ensure that this ‘rural-proofing’ of policy happens.
The committee’s overall vision was for balanced protection and promotion of the natural environment and a reversal of the biodiversity decline.
"This must be coupled with better recognition of the potential of rural communities and the rural economy," he said.
There must also be greater effort from the Government to ensure that policy changes do not work to the detriment of rural areas
Sign up to our newsletter to receive all the latest news and updates.