Virtual assembly is a promising application of virtual reality in design and manufacturing and has drawn much attention from industry and research institutes. Physically based modeling has been an important research topic in computer graphics and virtual reality. In this paper, physically based modeling issues in virtual assembly are investigated. The specific requirements and characteristics of physically based modeling in virtual assembly versus those in traditional computer graphics are analyzed and studied. The mass properties of the assembly models are extracted from the Computer Aided Design (CAD) system while the design models are transferred from the CAD system to the virtual assembly environment. This added information allows the assembly models to be categorized using human strength survey data. The interaction of parts, environment objects, and the human are analyzed. In the fully immersed virtual environment, it is discovered that certain presentations of gravitational acceleration needs to be scaled down to achieve maximum realistic feeling. Finally the benefits and limitations of physically based modeling in virtual environments are discussed.