Stability properties of a physical system refers to how the system responds to some perturbation; whether the system can recover on its own after being perturbed or whether it goes haywire. Stability analysis helps us to understand what happens when we perturb a system.
A classic example of a perturbed system is a ball and hill. Let us assume that we have a ball sitting in a valley. On one side of the ball, there is a large hill that continues upwards very high. On the other side is a small hill but beyond that there is a drop that goes down to infinity.
When it's at rest within the valley, its height is stable:
What happens if we give up a small push up the side of the high hill (smaller than the height of the small hill)?
The ball rolls down the hill into the valley, then up the smaller hill, then back down into the valley and up the hill, etc. all the while losing energy to friction, travelling up the hills less high each time and eventually coming to rest back within the valley.
The height of the ball over time can be expressed as a sinusoidal pattern with an amplitude that decreases over time and eventually flattens. We say that for this small perturbation, the system is stable because it returns to a steady state.**
But what happens if we give the ball a large push up the side of the high hill?
The ball will roll down the side of the high hill, into the valley, up the small hill and past the maximum height of the small hill then continue up until it reaches a maximum height and then begins to fall into the infinite drop where it continues forever.
The height of the ball never reaches a steady state again so for a large perturbation we say that the system is unstable.
**NOTE: A continuously oscillating system may also be regarded as being stable.
Those are the basics. Stability analysis has a wide-range of applications in the real world. A project I once had involved performing a stability analysis on the rate of star formation in a model galaxy if the galaxy was perturbed by a massive cloud of hydrogen gas. This was modelled numerically using a set of ordinary differential equations that govern the rate of star formation. You can read the paper at my blog about stability analysis: http://stability-analysis.blogspot.com/