Passive Income to fund (an early) Retirement

Passive Income to fund (an early) Retirement

You’ve saved and built your nest egg over many years. A lot of it is probably in your tax-sheltered retirement accounts. How can you retire “early”, because there are penalties if you access your tax-sheltered retirement funds (401ks, IRAs) too soon? For example, the general rule is that you will be penalized 10% if you withdraw tax-sheltered funds prior to reaching age 59 1/2. What if you don’t want to wait until then?

There are a few exceptions or other ways to access tax-sheltered accounts before age 59 1/2, but another approach is to build “passive” income to replace your needs until you can access the tax-sheltered money. If I could go back in time, I would have started building our passive income sooner. We just started working on this in the past few years.

We are estimating a budget based on our expenses after we stop working, but before we can tap Social Security and tax-sheltered accounts. Keep in mind, much of our budget is dedicated to saving for retirement, so this substantial part of our “expenses” will go away, remember to “pay yourself first!” Ideally, we will time it to minimize or eliminate other big ticket items: mortgage, tuition, etc. Our youngest child will still be in school during our targeted retirement, but we plan to payoff our mortgage. Since I love cars, I will probably still have a car payment.

Dividend Income

We have a brokerage account with TD Ameritrade that we opened around 2000, but we did not contribute to it regularly until the past couple of years. In this account we focus on Dividend Growth Investing (DGI) and also Income Based Investing, while also trying to be tax efficient. Because this is a taxable account, we select tax advantaged investments, including ones that pay Qualified Dividends (QDI) or even tax-exempt divvies.

We make regular (monthly) contributions to this account as part of our budget. We are targeting a rate of 8+%, so $100K would generate $8k year without touching the principal. This return may seem high, but I mix my own research with the High Dividend Opportunity service on the Seeking Alpha platform. With COVID-19, murder hornets, and other debacles of 2020, it has worked out ok so far, but we’ll see how it tracks and continue to adjust as needed. This platform has provided exposure to several other types of investments that I had not used in the past. In addition to existing positions in common stock, MLPs, and mutual funds, we now have the following as well:

  • Closed End Funds (CEF)
  • Preferred Stock
  • Exchange Traded Funds (ETF)
  • Bonds
  • Baby Bonds

For us, the key to these investments is that they pay dividends, interest, or other distributions. The tax treatment varies. Some examples below:

TickerInvestment NameInvestment TypeTax Treatment* Yield
TAT&TCommon StockQualified7.08%
EVNEaton Vance Municipal Income TrustCEFTax Exempt4.33%
CDR-CCedar Realty Trust, 6.50% Series CPreferred StockIncome7.36%
GECCNGreat Elm Capital Corp., 6.50% NotesBaby BondIncome6.66%
* – Forward Yield from Yahoo Finance or Seeking Alpha as of market close 2/25/2021

Focusing on dividends and distributions has been a change from my early investment days. In the beginning, I would to “buy low” and “sell high”. Makes sense! With this approach, you purchase a stock, like Amazon (AMZN), when it’s “low”, trading at $250/share. You sell it later when it’s $2,500 a share, you made 10 times your original investment (a 10 “bagger”). Obviously, this works best when stocks appreciate a lot and the growth is called “capital gains“. You pay tax when you sell the stock and it’s classified as either short term capital gains (holding for one year or less) or long term capital gains (holding over one year). Generally, short term capital gains are taxed as “income”, but long term capital gains have a more favorable structure.

A dividend stock generally makes regular payments (usually quarterly, but could be monthly or semi-annually) to the party that holds the stock on particular date, one day before the “ex-dividend” date. For example, T (AT&T) pays $2.08 per share annually ($.52/quarter). I have 500 shares of T, so in a year I would get paid $1,040. I don’t have to do anything, just hold the stock. This is why it’s called passive income! You can see how this could be beneficial if you need income but you’re no longer earning a paycheck from “work”.

I can still sell shares for capital gains, but if T continues to pay dividends, I may not care too much whether the stock appreciates or not. In fact, if I reinvest dividends, which I do in my tax-sheltered accounts, I don’t mind if it trades “sideways” so I can get more and more shares and increase my eventual income stream. In fact, if one of my holdings has a large capital gain, it can make my “Yield on Cost(YOC) look great, but it may make sense to reallocate the capital gains and increase my overall income. It took me awhile to grasp this, even though I had heard the principle. It helps to write down an example to understand, such as this article I wrote on YOC.

Some companies regularly increase their dividend every year. For example, the “Dividend Champions” is a listing of stocks have have increased their dividends every year for 25+ years, including T! Buying stock in companies like this is often associated with the Dividend Growth Investing (DGI) approach. There is an even more exclusive club of “Dividend Kings” that have increased their dividends every year for 50+ years!

For our AT&T example, the tax treatment for dividends is “Qualified“. This is similar to the distinction between short term capital gains and long term capital gains. You pay a more favorable rate for qualified dividends vs non-qualified. Non-qualified are generally taxed the same as ordinary income. Qualified dividends are taxed at a lower rate with a structure similar to long term capital gains. It may make sense to hold these type of investments in a non tax-sheltered account, but you need to consider your own situation and can consult your accountant or financial planner for guidance!

If we look at the EVN (Eaton Vance Municipal Income Trust) example, the dividends are actually tax exempt! It pays a lower yield of “only” 4.33%, but since it’s not taxed, this is the equivalent to 6.46% yield based on our income! Not bad! I used this Tax Equivalent Yield calculator at to find the equivalent yield, so many great tools available online!

Since EVN consists of municipal holdings, it is fairly consistent with a 5Y beta of .08 per Yahoo finance. Beta is a measure of a stocks volatility or how much the price fluctuates, compared to the market. An index like the S&P 500 would have a beta of 1.0, since it is a good measure of the overall market. A beta > 1.0 means it is more volatile than the market, so .08 means it is not very volatile. This may not be a good equity to buy for growth or capital appreciation, but it fits well in my portfolio now from a “Capital Preservation” perspective. Look at the chart of EVN below for January 1999 to February 2021 from It looks somewhat volatile at first, but then note that the amounts on the y-axis are fairly narrow and over this long period of time, it has mostly traded in a range between $11 and $13!

EVN chart – Source:

When you’re younger, you have more time to “make up” for mistakes. You can generally be more aggressive. A stock with a higher beta means more volatility, which could be worse losses than the market, but more optimistically it could be higher gains. We have seen the huge growth in tech stocks like TSLA over the past couple of years. We have seen the insane volatility with Game Stop (GME) and other heavily shorted Reddit stocks. According to Yahoo Finance, the 5Y beta for TSLA is 2.09, lots of volatile growth! The same metric for GME is -1.95, ouch!

We will look more at different metrics and ways to compare stocks, and also different mechanics for other investment types in other articles. A lot of it may appeal to nerds πŸ€“ or those who like money. I do notice people’s eyes glazing over when I start rambling about some of the technical details…

Below is the max chart for TSLA for 5/25/2010 to 2/26/2021 from Look at the Y-Axis! This reflects a 5:1 split at the end of August 2020. I did have TSLA, but sold my last lot in January 2020 for a 435% gain, which sounds good until you see what happened after… πŸ™ There is definitely some “Fear of Missing Out” FOMO, but we still hold other tech stocks that have been big winners. In many cases, we are divesting them over time to sharpen our focus on capital preservation and income. It’s hard to part with all of them completely, and we will generally keep a small position “just in case”.

TSLA chart – Source:

For fun, here is a chart of Game Stop from 2/11/2002 to February 2021. Yikes! Definitely potential to get rich or lose lots of money. Don’t forget:

β€œthe markets can remain irrational longer than you can remain solvent.”

John Maynard Keynes
GME chart – Source:

Real Estate

I have often considered buying a rental property or a vacation home as an investment. I don’t think we have the temperament or frankly, the luck, to pull this off. I read about real estate crowdfunding online and signed up on the Realty Shares platform a couple of years ago, and it has been a mixed bag.

I had 6 debt investments on Realty Shares, 4 exited successfully, with a return of approximately 8%. The remaining investments are still pending, having run into problems even before the COVID-19 debacle of 2020. I received about 50% of the principal on one $5k investment. There is a forbearance agreement that expired, so we’ll see how much, if any, I get of the rest. The other one stopped making payments, but has since resumed and I’m cautiously optimistic that the sponsor will be able to refinance and close the deal successfully. This was a wake-up call and taught me to be much more cautious going in, the initial successes made me overconfident. In retrospect, these were “fishy” deals, including a 2nd lien loan. What was I thinking!? A phrase I heard a lot when I first started researching different investments and it is still relevant as ever: do your own due diligence!

Incidentally, you have to be an “Accredited Investor” to participate in the Realty Shares platform. As defined in Rule 501 of Regulation D of the SEC, a few of the ways for individuals to qualify include:

  • A net worth of $1,000,000 or more, excluding equity in the primary residence
  • An individual income of $200,000/year or joint income of $300,000/year for the last two years with the expectation of the same for the current year.

A few years ago we reached the $1M milestone excluding our primary residence, so here was a benefit that we could get out it! More recently we hit $2.3M net worth, which supposedly means we’re rich, but we know we have not achieved our goals yet.

I signed up for a similar platform called Realty Mogul that also required accredited investor qualifications. For Realty Mogul, I had an interview and shared some paperwork as backup. With Realty Shares I just had to fill out a form online, which seems a little sketchy in retrospect. Hopefully this is a sign that Realty Mogul is the real deal, but so far been to scarred to pull the trigger on any deals. Keeping an eye on how things shake out with COVID-19 and the economy before giving crowdfunding another try.

We switched to the Fundrise platform since the fall of 2019. I have been happy with them and their improved quality, transparency, and communication. Fundrise outperformed the markets and REITs over the first half of 2020, with a return of roughly 5%. This matches what I have seen in my own account, but generally the return has been closer to 10% over the past several years. So far, it’s nice to have some exposure to real estate for more diversification, without the pain of managing a physical property. We also contribute to Fundrise on a monthly basis as we try to build our passive income.

Where are we now?

SourceAccount TypeAnnual Amount
Roth IRATax-Sheltered$10,000
Roth IRATax-Sheltered$10,000
BrokerageNon Tax-Sheletered$7,000
FundriseNon Tax-Sheletered$1,000
Estimated dividends and interest for the next 12 months as of 2/2021

Our passive income across all accounts overall is about $70K/year, but out non tax-sheltered passive income is only generating about $7K/year. This is growing as we continue to contribute to it, but we need to bump this up significantly! Our target retirement adjusted budget will be less because we won’t have the following items:

However, we are still short of the goal if we wanted to rely on Non Tax-Sheltered income. Achieving the target monthly budget post-employment in taxable accounts is a daunting task. We have some other ideas: Maybe I can start a few side hustles to earn some level of income to offset the gap. Hmmmm, maybe I can start a blog? πŸ™‚ I’m 47 years old now, looking to retire in the next couple of years, so we have some time to figure it out. We can also look into ways to tap our tax-sheltered account before age 59 1/2, ideally using methods that don’t impose the 10% penalty. Some options include:

  • Building a Roth Ladder
  • Take SEPP through 72(t)
  • Use the Rule of 55
  • Just pay the 10% penalty

Subscribe to get email updates when we post new content and please discuss further in the comments below or let me know if you have any questions or other feedback.

~$Retirement NerdπŸ€“

Disclaimer: I am not a financial planner and content on this site is meant to provide food for thought, not professional advice. I share my experiences to show what worked so far and what didn’t, YMMV. Please consult your financial advisor or tax professional as needed.

By $Retirement Nerd

$Retirement Nerd is in his late 40s and looking to retire in the next few years.


  1. After looking over a handful of the articles on your site, I honestly appreciate your way of writing a blog.
    I book marked it to my bookmark webpage list and will be checking back in the
    near future. Take a look at my web site too and let me know your opinion.

  2. This is really great article. Building passive income, it seem getting harder Over time. Therefore I guess not for everyone, I have personally try it myself. Like blogs, have little bit success with it.

  3. Hey! I just wanted to ask if you ever have any issues with hackers?
    My last blog (wordpress) was hacked and I ended up losing many months of
    hard work due to no backup. Do you have any solutions to stop hackers?

    Here is my web page gold bee

    1. Hi,

      WordPress is such a popular platform that it is definitely a target for hackers and spam. I get A LOT of spam, haha. πŸ™‚

      One approach is to try to follow best practices and update software (themes, plugins, etc.) regularly since they patch vulnerabilities. As part of a holistic approach, as you discovered it’s important to have a regular backup and recovery plan in place.

      Good luck!

Leave a Reply

Your email address will not be published. Required fields are marked *


Enjoy the $Retirement Nerd Blog? Connect or Share