Mate I can relate to your story, although mine didn't end up with as much damage......
when I first got the mighty HMAK MARISTIKA..... one of my mates (non-forum member) said lets hit up "rays rock" which I now know to be Brays.... so we made our plan.... the wind forecast was fine..... swell on the other hand, well I thought 3-5m hmmm thats not too big is it...... anyway drive up there..... have a look at the waves....... think wow.... that looks way bigger than I thought.... and it is really dumping hard (at Moffats)...... hmmm lets look elsewhere...... around to the other nearby beaches including kings etc.... hmmmm nah, they look a bit worse....so lets rig for launch anyway........
Get down on the beach, the waves look nastier there......hmmmm not really sure I should do this.....those bad ass dumping waves don't look real friendly..... hmmmm Oh well I have driven all this way..... have a PFD, am reasonably fit.... worst that can happen is.... i get really wet..... and turn the yak upside down.......everything is tethered..... so lets give it a crack.......
Well........ the ride out was 'ok' in that I pushed it into deep enough water to pedal..... jumped on and went to pedal...... hmmm nope pedals not in properly...... cant use feet, am now furiously pedalling by hand...... the PA punched through the waves like the aircraft carrier that it is..although lit was as though it was on a scale..... I.E equivalent to sea state 4 in a real carrier! waves were pouring over the front, and I was thankful of the warmness of the QLD ocean..... and that I was not in the Arctic! and I was soon over the back of the waves......sorted the pedals out.... and it was sweet.... thought, bloody hell not looking forward to coming back in through that.....
So out goes the trolling rod... we start heading towards the reef.... fishing around.... get nothing, I am still trying to work out how to use a sounder.... (truth be told that journey is still incomplete!) get over near Brays.... massive waves are breaking over it.... hmmm stuff that, don't get too close.....hey whats that, Tuna bust up..... I am off, chasing them towards kings beach..... end up pedalling miles... wow it is easier than i thought.... oh well better turn around..... damn that is why it is easier..... that is the direction of the current! dammit!
A slow trip back towards brays.... catch up with my mate, and he is all wet.... what happened? Oh i got a bit close to the reef, and a massive wave smashed me...... lost a few things.... anyways we keep fishing a little bit.... then call it a day.... a massive wet Donut......
so the slow......slow journey back towards Moffats against the current..... get there.... stow all items..... watch the sets.... think damn, this is definitely going to go pearshaped! hang on, there is a really calm bit....... as i pedal in, it is a crazy Rip...... and I am literally going at a very very slow walking pace........but, no waves, it is sweet, it took about 20 minutes to get in ..... and about 30m from the beach I heard a noise, turned around to see a massive breaking wave about 4m from the yak.... Oh shit....... yup, PA sideways, me out of it, it careening towards the beach.... upside down... me tumbling and surfacing..... swim the rest of the way in......grab the yak, turn it up the right way.... drag it onto the beach and just lay down absolutely shattered!
a few minutes pass, realise that the only thing lost was my spyderco knife, that was on my PFD un tethered.....
cart in, drag yak back the the car..... pop the bungs..... wow that forward hatch does let in some water...... a few litres pour out......
moral of the story..... tether everything, and well have bloody fun regardless

But if it is too big, then unless you are keen to get very wet, lose some stuff, and damage things, head to the shelter of the estuary..... but I will not go out in such crap conditions again that is for sure! I still enjoyed it, and learnt a lot.....