Originally, there was no "cave" as such; in Detective #33, Bruce slips into his "secret laboratory": The concept of hidden rooms and sliding panels was a staple of boys' fiction from Edgar Rice Burroughs through the Hardy Boys and beyond.
In Detective #63 we learn that Batman parks the Batmobile in the barn located on his property and then enters Wayne Manor through an underground tunnel.
The next time around, the only thing we saw was the Trophy Room, and it certainly appeared to be located above ground: That's from Batman #12 (August-September 1942). Later, in that same issue, we got the very first view of the cutaway of Wayne Manor, showing the underground "hangars": As you can see, it's just a glorified basement, with no indication of stalactites or bats.
In Batman #16, Alfred stumbles upon the sliding panel that leads to this:
In the Batman serial from 1943, the second episode was entitled The Bat's Cave. Batman and Robin bring a crook back there and interrogate him, with the bats flying about so unnerving the criminal that he tells everything that he knows.
Finally, in Detective #83, we got our first real look at the Bat-Cave in the comics: The intruder in the Bat-Cave turns out to be their butler, who's exercising. Yep, this issue also saw the debut of the slender Alfred.