The original house was rebuilt in 1829 by William Pugh (1783-1842) of Berriew, financier and philanthropist, who supported the extension of the Montgomery Canal (1821), the building of Dolfor Road (1823), the introduction of steam power to Newtown's mills (1833), and who was a benefactor to the poor of Newtown. After his financial failure, exile and death in 1842, the property was acquired in 1841 by John Naylor of Leighton, the gardens becoming well known for their planting and giraffes looking out from the menagerie. The house was substantially extended for Christopher Leyland (Naylor), his son, in 1887. The house was sold in 1919, and subsequently became a school.

The 1829 house is in an Italianate style, with white painted and pebble-dashed brick walls, and shallow pitched slate roof with wide bracketed eaves. Triple round-headed windows of this building survive at the rear, and at the canted end of the rear wing. The 1887 extension at the front is of 3 storeys, yellow brick with limestone dressings, and slated roofs. This comprises a rectangular block, 3-bays x 2, with narrower 3-storey protruding bay the front having a semi-circular bay of two storeys carrying an open belvedere with a swept lead roof and crown carried on rustic cast iron stanchions. The NW corner is enclosed in a raised and glazed verandah carried on similar Beaux-Arts style ironwork with elaborate brackets covering the main double oak entrance doors.

Under Welsh Government's 21st Century Schools Programme (Sustainable Communities for Learning) there are plans to construct a brand new, purpose built special school (Brynllywarch) for pupils with behavioural and emotional difficulties at the existing site in Kerry. Cost £9.1m. The developer, ISG Ltd, was appointed in June 2022. 

References: Cadw listed buildings database; 'Contractor appointed for £9.1m Brynllywarch Hall School project', Powys County Times, 13 June 2022
RCAHMW, 2010. Updated by M. Powel, RCAHMW. December 2022