It sounds like he could be lethargic from cool water temps. Bettas really like water to be in the low 80's. Sounds like your water changes are good - are you doing 100 percent changes? With a 3 gallon tank you could do an 80 percent or so as long as you're getting most of the gunk from the bottom of the tank, too. Bettas like still water, so not having a filter is fine. The brown color you're describing could be natural coloration. It's hard to tell without more description or a picture.
As far as depression, every betta is different. Mine will always swim up to me to say hi when he sees me. You could try moving him to a more lively area of your home. But I would definitely try giving him more heat - and a heater would be more efficient in a larger tank. A 5 gallon would be perfect although 10 gallons are more common and are a great price plus all of the accessories that you need are cheaper.

Hope some of this helps!
Let us know how it turns out!

Get your betta a heater and a thermostat. Your fish is probably too cold. Bettas are very tropical fish - make sure they are warm! Get a heater that heats a 2.5 to 5 gallon aquarium.

Your size tank (3 gallons) is pretty sufficient for your betta, that is enough room.

TammyPie