There's a good article here on the difference between the System Cache Plugin and the Global Configuration Cache.
It's best to leave all caching disabled while developing.
If you make changes and don't see them it could be because caching is enabled.
The cache can be cleared at
Site -> Maintenance -> Clear Cache
Caching is usually only useful for sites with lots of content and visitors, but might help some extensions and templates load faster. Give it a try. If you don't notice a difference in page load speed disable it.
