The earthworks of the medieval castle occupy the southern end of a steep-sided inland promontory and are thought, though not proven, to have utilised an earlier prehistoric inland promontory fort; one of a number situated along the river to Sandyhaven Pill. Tradition also associates the site to the grave of King Arthur's nephew, Gawain.

The medieval castle was situated in the Cantref of Rhos, an area settled by the Anglo-Normans from 1093, brought under control by Roger de Montgomery's son Arnulf from his base at Pembroke. This area was subsequently reorganised as the Lordship of Haverford, initially subject to the Earls of Pembroke. From 1247 onwards, the Lordship also contained the large barony or sublordship of Walwyn's Castle, commonly independent of Haverford and for which the castle here was the main stronghold and administrative centre. By the mid thirteenth century the castle had been acquired by the de Bryan (de Brian) family, whose main seat was at Laugharne and thus the castle never further developed and remained a predominantly earthwork castle throughout its existence.

The earthwork remains comprise a 5m high curving bank that crosses the neck of the promontory on its north side, cut by a 4m wide entrance. Slight traces of an external ditch survive, as does an outwork surrounding the entrance that defines a circular area 56m in diameter. Traces of stone and the outline of a probable curving structure survive here. The interior area of the castle is an irregular roughly tear drop shape, 1.3ha in extent. On its east, south and west sides it is predominantly defended by steep natural slopes with the exception of a 75m stretch along the west side which has been artificially enhanced. It is unclear whether a similar although much slighter enhancement along the east side also relates to the castle or is of a later date. The castles interior is divided in two by a straight section of bank and ditch running across the promontory's narrowest point. This stops short of the western edge of the promontory and thus provided an entrance into the lower bailey. At its eastern end the dividing bank curves outwards to create a roughly square motte nearly 6m high by 13m wide. A depression on top of the motte is likely to mark the site of a circular tower and here there are also traces of stonework, which may indicate both a stone tower as well as a wall along the top of the bank. The only other extant remain within the interior is an earthwork running southwards and dividing the upper bailey, most likely representing a later field division.

A detailed survey of the castle was carried out by RCAHMW and the Walwyn's Castle Historical Society in 2009.

Louise Barker, RCAHMW, Jan 2010.