Exactly. It's all about perspective.
The financial part of me looks at it like this:
I buy this home. I go hard on the payments during the first two to three years to achieve at least 50% equity in the home. I then look for a $200K - $250K home. Sell your home, guaranteeing a net of at least $50K. Put down the $50K, which will be 25% to 20% of the total which enables you to forego PMI and now your mortgage is is close to the payment you were paying for the $100K home. Rinse and repeat in a few years with a $300K home. You could even consider keeping the $100K home and just leveraging the equity to buy the second home while renting the original home.
You have to think about the long game brother. The start will always be less than optimal. But, staying the course will yield positive results.