My half-retired in-laws live part-time 5 minutes from "Party Rock", and I've been spending many many weekends up there for 8 years. I know the river quite well - I have kayaked every section of the Toccoa (from Deep Hole to the dam on the lake) numerous times. My kids love to swim here, and I've save several coolers from flipped canoes as well

There is no other spot on the river like that location for access by so many people for sport and recreation, and it is a common spontaneous "tourist spot" where folks just stop the car and jump out and take pictures.
I know my kids (ages 5 and 9) will miss swimming there.
I've spent enough time "hanging out" with the kids to understand why the local landowners may wish to restrict access. But a few bad apples ...
As more property gets developed, this points to a bigger issue -- this section of the Toccoa REALLY needs a public access point (the USFS "Canoe Trail" ends at Sandy Bottom - the last spot with easy public access is along this stretch above Shallowford Bridge). I don't fish, but I've spoken with trout fisherman who complain about similar issues (in fact, while our uses are different and at times conflicting, fisherman and rec/whitewater boaters share a lot of common issues).
From what I can tell from topo/hiking maps, isn't much of the "Turkey Farm" tract just downriver public land? It is sometimes used for the Aska Adventure Race, and I've parked at the end of the road and hiked down to the river there before. Right now it is a long or steep hike to the river and/or a bushwack.