A fun and less expensive way to collect coins is by "type" rather than by date. That is, rather than trying to find or buy a complete set of Lincoln Cents, you might have ONE Lincoln cent that would represent the whole series. Of course for U.S. coins you'd also have an Indian Head Cent, a Flying Eagle Cent and a Large Cent.
That would be a basic "type set" of U.S. cents and needn't cost a lot of money if you are content to have circulated examples.
Most type set collectors would take it a bit farther. For example, Lincoln Cents had a "wheat" reverse until 1959, at which time it changed to the Lincoln Memorial. You'd probably also want the four special reverses of 2009 and the "E Pluribus" design that started in 2010.
Most would also include one of the "steel cents" of 1943. These were made to conserve copper for wartime use.
That would seem to cover the major varieties of Lincoln cents, but one more is often added: that's the 1909 issue with "V.D.B" under the wheat ears on the reverse. Those were the designer's initials - Victor D. Brenner, but many people objected to him "signing" so prominently, so later that year his initials were moved to a less visible location under Lincoln's shoulder (tip a penny and you will see them there still - though you may need a magnifier!).
By the way, our one cent coins are technically "cents", not "pennies". Some folks make a big deal about that.
There are a few more minor varieties to consider. In 1944, and again in 1983, Lincoln Cents changed their composition. Lincoln cents were bronze (95% copper, 5% tin and zinc) from 1909 to 1942, steel in '43, without tin from '44 to '58 (though 1944-1946 are more brass) and then with zinc and tin again until 1959. In 1962 this was changed to 97.5% zinc with a light coating of copper - which is why modern pennies get really ugly looking very quickly.
You could carry this much farther. There are varieties where the first two letters of AMERICA are nearly touching or not - some would make sure to include examples. There are also large and small date varieties of 1960 and some other years and the famous "double die obverse" of 1955 (and other years).
I've only talked about Lincoln cents - there are multiple varieties of Indian cents, Flying Eagle cents and Large cents. There are also "patterns" - transitional coins never intended for general circulation.
When you start including older coins like the Large Cents in a type set, the cost can start to mount. Few of us could afford a 1793 "chain" cent in anything but the poorest condition and even that could run thousands of dollars.
I used a holder much like this for my first Type Set
Other ways to collect coins
There are also "Commemoratives" - coins issued for special events, like the 1893 Columbian Half Dollar and even that 2009 "Log Cabin" cent. Many would include some of these in a type set.
There are Civil War Tokens, Colonial Coinage and coinage from "private" mints - and that's just the United States. British coinage goes back much farther and Roman coins can be older still and yet still be common enough for collectors.
If we start collecting nickel and dime types, quarters, halves and dollars, as well as the odd two cent and three cent pieces, you could spend a fair amount of money amassing even the most basic set.
As to gold coins, that really starts to get expensive - even without considering the rare types and varieties.
However, collecting by type is really up to you. You might not want to collect anything before some arbitrary date - your birth year, for example. Or you might limit your collection to one series - Lincoln cents or Jefferson nickels. It's up to you.
Here's an idea: a "Coins of the world" type set, or even a "Copper coins of the world". This could be great fun and very educational.
How about "Coins with famous people" or "Coins with animals"? Or "Coins of my birth year"?
Very specialized collections might be "Coins made at the San Francisco mint" (that one could set you back a few dollars!).
The possibilities are only limited by your imagination and of course your wallet in some cases.
Note: All my coins are in a safe deposit box. I keep nothing in my home.
Contests and giveaways: http://coins.aplawrence.com/2014/05/contests-and-giveaways.html