First of all, what do you mean by "closed system?" Some individuals are interested in biospheres that have absolutly no holes or vents and are completely self-sustaining (for a while).
I am assuming that you are talking about a typical terrerium. If so, there is absolutly no way that there would not be enough oxygen unless every single nook, cranny, and hole were closed. Remember, reptiles, amphibians, and especially insects have completely different circulatory systems than humans and have low oxygen requirments to begin with.
In regards to ventilation, what size is the terrerium? Most likely, you don't need any sort of fan driven setup. If you are using a glass aquarium, which is probably the best choice for a beginner, the best way to control temps/humidity is to experiment with the top. Use glass and screen to completly cover the top. As for the ratio of screen to glass, experiment with it to find the best temp/humidity balance. You should probably place your light over the glass section of the top.
Regarding insects escaping, that shouldn't be a problem as long as you use a fiberglass screen (from somewhere like Home Depot).
Honeslty, a healthy, balanced terrerium doesn't produce lots of bad odors if it is maintained, and, of course, built correctly in the first place (don't skimp on the research!). It is very important to have the right balance of humidity/temp/light/plants/# of animals. Generally, keep the # of animal inhabitants to a bare minimum, and pack in as many plants as possible (in good taste, of course). As long as your plants are healthy, they should absorb the materials that create odors. However, gardening-grade charcol, when mixed with soil, will absorb any bad smells.
As I recall, you seem interested in building a pond or water feature in your terrerium. This is not to be approached lightly!! Honeslty, the only way to have a really nice water/land setup is to use a "false bottom"~research that on this forum.
One more thing, plants purify and refresh air! I wouldn't worry about a fancy air filter.
Good luck with your terrerium! Be sure to search the forum and take advantage of all the great info it provides.
If you want a really basic setup plan (no water feature), you can visit the link below.

Here is a link that might be useful: basic terrerium setup