I have just uploaded two projects to github that in my oppinion solve the problem with using ESP8266 modules as slaves to Arduino MCUs.
Instead of the serial interface the SPI interface is used here. Therefore the ESP module must be reflashed to a custom firmware.
The WiFiSpiESP project is in fact an application and must be flashed to ESP module while the WiFiSpi project is an Arduino library. I tried to keep the library objects and functions as close as possible in its behaviour to the default Arduino’s WiFi library in order to make migration to the WiFiSpi library easy.
I know that flashing ESP module may not be a solution for beginners, but I am afraid there is no chance to make a robust solution on AT command protocol and SoftwareSerial class.
The projects are: