How often do you have to solve a small problem that is a part of a
larger project and decide to take the time to perform some seperate
experiments to solve the problem, before adding the partial solution to
the whole? In the past, I hardly ever did that and everytime I
encounter such a situation, I still have to resist the temptation to
take the route that seems to be the shortest, but that has long proven to be the longest road to the solution.
As an example, suppose you need to extract some information from a string and decide the best way is to use a regular expression. You know that the problem isn't trivial and that you probably need a few attempts to get it right. Still, your initial inclination is to just put the regex in there and go through the motions of compiling the code, performing the steps required to invoke the code (click a button, enter some text, etc), find out that the regex doesn't work, modify it and go through the motions again and again and again, probably while making some other modifications. Because the cycles take relatively long, progress is slow and you start cursing every typo. If you recognize this, then I think you can become faster and happier at your problem solving by reading on.
The advantage of the 'intuitive' approach is that, if the solution is right, you have immediately come closer to solving the larger problem. However, if the solution fails, the iterations to be informed of your failure and to verify subsequent modifications may soon take more time than developing a seperate solution to the subproblem would have taken. For some reason, we always seem to underestimate the amount of work it takes to get it right and consequently we opt for the small change that we will have the correct solution the first time around.
Some of you may now be thinking that the obvious solution is to write a proper unit test and run that unit test after every modification, until the code passes the test. I agree that goes a long way, but usually the test is part of a larger number of classes and running all those tests takes times, especially if it bootstraps an entire Spring-Hibernate application or something of the like. In such as case, it still takes more time than is needed.
My solution is not to be afraid to experiment. It seems to cost too much time to create a new script or class, seperate from the larger project, provide the correct libraries, run this small project multiple times to get feedback on your solution and finally copy-paste the solution back into the project your are working on. However, my experience is that it is well worth the time you need to do this, because it prevents the endless cycles of building-deploying the entire application and getting it to provide feedback. I hardly write a regular expression or SQL query without first testing it seperately from the application. Now these example are given at the smallest level, but it also holds for somewhat larger design issues and even for issues people call 'software architecture'.
Another thing about these small experiments that I find a major advantage, is that it gives you the freedom to explore some avenues that you wouldn't dare put into the actual project. You can't just start switching libraries or refactoring relatively large parts of the code and changing too much is risky and frowned upon, even if you can easily revert your changes (assuming you are using a versioning system. If you aren't, stop reading, install Subversion and commit your code before continuining this article).
A third advantage of experimenting is that it encourages you to rewrite and polish the code that you are writing. You can go through more iterations in experiments 'outside' of the project, because you have a clear overview over the code involved. Modifications of code inside a larger project are less inviting to proper refactoring to do 'the right thing'.
Painters, writers, craftsmen, even philosophers: if famous ones are asked for the secret of their succes, they always advise exercise and experimentation. I hope I have explained that it also simply makes sense. If you don't want to take it from them, take it from rationale.