I'm glad someone mentioned this. It's a popular myth that needs debunking. There's fairly conclusive archaeological evidence that shows the 4th and early 5th centuries were quite economically productive ones for the Empire with new towns still being established, particularly in the East, thanks to the relative peace on the Persian front.
The Third Century, in overall economic terms, was a far more trying period in Roman history.
Mike Duncan said it best "The question isn't why the Roman Empire fell, but what didn't it fall earlier".
If the Empire could survive the Crisis years thanks to inspired leadership under the Illyrians, then it can survive anything. Unfortunately, there was no Aurelian or Diocletian in the 5th century to save the Empire. That man didn't come till Justinian I, by which time it was already too late, two entire generations had grown up since the fall of central Imperial authority.