This is part of the scenic Cave Point County Park. Exploring the shores of Door County (a peninsula) you find a lot of the stacked rocks you see on the beach. Cave Point is full of low bluffs and water swirling under the rock.
The rock piles are called cairns. These have become more numerous and higher over the years now there are some restrictions.