Following the Norman invasion of Britain in 1066, William the Conqueror subjugated all of England under his rule. However, pushing into Wales was a much more difficult task, and so a region was created and fortified along the border known as the 'Welsh Marches'. William de Londres, a Norman lord who came over with William the Conqueror, was placed in charge of the Lordship of Gower, land which had been wrested from the Welsh realm of Deheaubarth. It is believed that a native Welsh fortification once stood near the present site of Oystermouth Castle, followed by a wooden structure built by the Normans.
Gruffydd ap Rhys ap Tewdwr, Prince of Deheubarth, attacked and retook Gower in 1116, burning down the wooden fortification that the Normans had built. The Norman retreat from Gower lasted until Maurice de Londres, William's son, re-established the de Londres family in the Manor of Oystermouth in 1128 and began construction of a stone fortified manor house around 1140. This manor house is now the South Keep, the oldest part of the castle.
Following the dissolution of the male line of the de Londres family, the castle was given by King John to William de Braose in 1203. William de Braose constructed a nearly identical building against the wall of the original manor house, creating the North Keep. The first floor of the North Keep would have been used as a 'Solar', the private residence for the Lord of the castle and his Lady. A massive back-to-back fireplace connects both keeps, which would connect to a single chimney. A basement below may have been used as wine or ale storage.
An additional multi-storey building and large tower known as the Northwest Range was added to the castle. Vaulted basements used for storage were also added, with one believing to have held a large water tank used in lieu of a more traditional castle well.
The castle was fortified with large curtain walls which surround the courtyard and a gatehouse. The design of the gatehouse would have accommodated the construction of two large drum towers on either side of the castle's entrance, but it is unknown whether or not these drum towers were ever actually constructed. Of particular note during this period is the construction of the Chapel Block during the late 13th century, which houses Alina's chapel. The addition of buildings such as this during the castle's latter construction demonstrates how it became less of a defensive structure and more of a luxurious residence over time.