Any time you have jargon or character motivations that can only be understood with an existing thorough knowledge of the world of the story, that’s Inside Baseball.
This isn’t to say that you don’t want to place your story in a specific world, or that you should shy away from telling stories that take place outside of the normal experience of the majority of people. It’s a question of balance.
There are ways to ease an audience into a specific world. Consider the Audience Surrogate. Ever notice how many TV pilots involve a New Person being introduced into an existing world, and how people need to explain things to this New Person?
It’s because we, the audience, need this information. If we’re going to follow anything that’s happening, we’re helped by having somebody inside the story that needs to find out the same information we do.
This doesn’t mean that a Surrogate should be on the receiving end of pages of expository dialogue. The difficult task is finding clever and sneaky ways to hide information for the audience inside the action of the story, like sticking a disgusting chewable pill in a spoonful of peanut butter.
The larger point, though, is never take for granted that your audience will have the same level of understanding of a specific world that you do. In fact, you should know more about it than they do. But, as you tell your story, you need to remember to let the audience in on the workings of this world. Be it a mythic kingdom or the trading floor of the stock market, the audience needs to be able to follow the rules if they’re going to have any hope of paying attention to the specific story you’re trying to tell.