We first learned of Apple's Head Mounted Display R&D project nearly two years ago to the month. Our first report titled Apple Preparing a Cool iPod Visual Head-Display System covered their preliminary work on their laser based binocular near-eye display and light engine systems. In today's patent we see that Apple is moving this project further along with a first overview of where the project is headed – in terms of ideas and basic objectives. Currently Apple is working on a couple of methods of making this work, including the concept of sliding a smaller iPod or even a smaller iPhone Nano into a dedicated frame which Apple views as a unique docking station of sorts. The Head Mounted Display system may contain its own video camera or accommodate a camera that is already integrated into your iPod touch or iPhone. The system will have some physical controls but other controls may be set off by voice commands and/or head movements that sensors could recognize as distinct commands. The system will likely offer a Picture-in-Picture (PIP) feature which I think is awesome. While you're viewing a movie, TV show or playing a game, you could always have your PIP set to the external world so that you could be aware of someone approaching you. The system will also offer advanced haptics in their earplugs area so as to enhance Movie sound effects. With Apple hiring a wearable computer guru of late – I would expect that this project could get fast tracked. Time will tell - but it sure looks promising.
The Face Plate
The all important faceplate 302 could be formed from any suitable material or materials. In some embodiments, the faceplate could be formed from lightweight materials that afford user comfort (e.g., plastic) while maintaining strength to support a portable electronic device. In some embodiments, the faceplate could be formed from a material capable of withstanding impacts or shocks to protect the components of head-mounted device 300. Examples of materials include composite material, glass, plastic (ABS, polycarbonate), ceramic, metal (e.g., polished aluminum), metal alloys (e.g., steel, stainless steel, titanium, or magnesium-based alloys), or any other suitable material. In some embodiments, the outer surface of frame 302 can be treated to provide an aesthetically pleasing finish (e.g., a reflective finish, or added logos or designs) to enhance the appearance of head-mounted device.
Camera Possibilities
The head mounted device may include a camera region 306. The camera region may represent a camera that is integrated with the head mounted device and/or it may represent an opening for a camera. An integrated camera may be used in place of or in conjunction with a camera on the portable electronic device. As example, the camera could be used if the portable electronic device doesn't include a camera of its own (e.g., iPod Touch).
In cases where the portable electronic device includes a camera, the front panel could include a camera hole, which could be aligned with a camera of the portable electronic device when the portable electronic device is situated inside head-mounted device. The camera hole can allow the camera on the portable electronic device to capture image based content of the user's surroundings. While a hole is not shown, it should be appreciated that it can be placed at any location on the front panel.
Exterior Controls
In some embodiments, the head-mounted device could provide controls 318 which could allow the user to control the portable electronic device while wearing the head-mounted device. Controls 318 could control any suitable feature and/or operation of the head-mounted device and/or the portable electronic device. For example, controls 318 could include navigation controls, display controls, volume controls, playback controls, or any other suitable controls. Controls 318 can be located on the side surfaces, front surface, top surface, eyeglass temples 312, or any other accessible location on the periphery of the head-mounted device.
In one embodiment, the head-mounted device may include one or more controls that are functionally mapped to the buttons of the portable electronic device when the portable electronic device is coupled.
Using a Remote with the Head-Mounted Device
The communication interface provided by a wired and/or a wireless connection could allow the user to control a portable electronic device using controls provided on a head-mounted device or a remote control.
The remote control 322 which is noted above could be connected to a head-mounted device or the portable electronic device using any suitable approach. For example, the remote control could be a wired device that is plugged into a connector in connectors 320. As another example, remote control 322 could be a wireless device that could transmit commands to the portable electronic device and the head-mounted device via a wireless communications protocol (e.g., Wi-Fi, infrared, Bluetooth or any combination thereof).
The user may use the remote control to navigate the portable electronic device and to control the display, volume, and playback options on the portable electronic device. The remote control could include any remote control device, such as, for example, the Apple Remote.
Utilizing Haptics
Haptic devices can interface with the user through the sense of touch by applying mechanical stimulations (e.g., forces, vibrations, and motions). For example, while a user is watching image based content, haptic devices can be configured to provide an enhanced surround sound experience by providing impulses corresponding to events in the image based content. As an illustrative example, the user may be watching a movie that shows an airplane flying on the left of the screen. Haptic devices can produce vibrations that simulate the effect (e.g., sound effect, shock wave, or any combination thereof) of the airplane. For example, a series vibration may be provided along the left temple from front to back to simulate the airplane flying to the left and rear of the user. Speakers can also be used in this manner.
Head Mounted Device may use an iPhone or iPhone Nano?
Apple's patent FIGS. 10A and 10B illustrate an exemplary portable electronic device (an iPhone) that may be used in any of the head-mounted devices described herein. It is particularly applicable to the embodiment shown in FIG 3A. In this embodiment, the portable electronic device is a handheld electronic device. Thus, it has a small form factor that can be easily managed by a head-mounted device. The key here that could be easily overlooked in that simplistic statement is that patent figures 10 A and B are revealing illustrations of an iPhone and not an iPod touch. Therefore what is being described here is more likely to be that of an iPhone nano type of unit. .
A head-mounted device may include a complimentary connector such that the head-mounted device and the portable electronic device can be operatively coupled when the portable electronic device is mounted relative to the head-mounted device.
The Slide Mechanics & Special Eye Sensor Adjustments
Apple's patent FIG. 4 shows a configuration for sliding a portable electronic device 400 into a head-mounted device.
In the implementation shown in FIG. 4, the portable electronic device 400 is slid through the slot 340 and into the cavity 332. The slot 340 may be situated opposite a connector 342 such that as the portable electronic device 400 is slid into the cavity 332, a connector 402 of the portable electronic device 400 can begin to mate with connector 342 and can completely mate upon final insertion of the portable electronic device 400 into the cavity 332.
The connectors 402 and 342 can allow data and/or power communications between head-mounted device 300 and portable electronic device 400. The connectors 402 and 342 may be widely varied, such as, for example, low profile connectors, USB, FireWire, or a 30 pin connector used in iPods and iPhones. In some cases, the cavity/connector combination may generally define a docking station for portable electronic device 400.
After coupling the portable electronic device 400 to the head-mounted device 300, the protocol under which devices communicate may be widely varied. Any suitable communication protocol may be used, such as, for example, a master/slave communication protocol, server/client communication protocol, peer/peer communication protocol, or any combination thereof. For example, using a master/slave communication protocol, one of the devices, the master device, controls the other device, the slave device. For instance, the portable electronic device 400 may become a slave to the head-mounted device 300 such that the head-mounted device controls the operation of the portable electronic device once they are coupled.
Alternatively, the head-mounted device 300 could serve as a slave of the portable electronic device 400 by simply implementing actions based on controls from the portable electronic device 400.
As should be appreciated, each user's eyes are oriented differently. For example, some eyes are located close together while others are more spread out. Thus, either device may include sensors for detecting the distance to the eyes and the position of the eyes. After determining the optimal viewing positions and size of the displayed image frames based on the detected distance and position, the head-mounted display system can adjust the viewing positions. By way of example, the portable electronic device and/or the head-mounted device may include cameras that can reference where the eyes are located relative to the head-mounted device.
The resolution of the displayed image frames could also be adjusted in a similar manner. However, because each user's eyes focus differently, it may be beneficial to allow the user to manually adjust the resolution as this is a more difficult measurement to make.
Clip Mechanics: Clip Media Player to Head Mounted Device
Apple's patent FIG. 7 shows a side view of a head-mounted device coupled to a portable electronic device. Head-mounted device 700 could include eyeglass temples 702, which allows the user to wear head-mounted the display system by putting the eyeglass temples over his ears. Thus, instead of sliding the portable electronic device 704 into the head-mounted device 700, as shown in the previous embodiment in FIG. 4, the portable electronic device can be coupled to the head-mounted device using clips.
In some embodiments, head-mounted device 700 can include top clip 706 and bottom clip 708 to hold portable electronic device 704 in place in front of the user's eyes. In the embodiment shown in FIG. 7, the screen of portable electronic device 704 may face the user's eyes.
Clips 706 and 708 can be formed from any suitable material, such as for example plastic, rubber, and foam. In some embodiments, clips 706 and 708 can follow the contours of portable electronic device 704. The user may place portable electronic device 704 into head-mounted device 700 by snapping portable electronic device 704 into bottom clip 708. The user can then rotate portable electronic device 704 until the rest of the device fits into clips 706 and 708 (e.g., following an elastic deformation of at least one of clips 706 and 708). In some embodiments, the user may slide portable electronic device 704 into clips 706 and 708.
Apple's patent FIG. 8C shows a head-mounted device that has an adjustable frame. In the embodiment shown in FIG. 8C, the frame of the head-mounted device could rotate toward the user. For example, the head-mounted device could have pivot bearing 806 (e.g., hinges) connecting the frame and eyeglass temples 808. Pivot bearing 806 could allow the frame to rotate around the eyeglass temples. As a result, the head-mounted device could serve as a display stand for the portable electronic device if the user decides not to wear the head-mounted device.
As shown in FIG. 9B, the front space 918 could be broken up into a variety of zones. In this particular implementation, the front space includes three zones 918A, 918B and 918C. Each of these zones can include a variety of electrical and mechanical components including but not limited to batteries, processors, circuit boards, cooling systems, memory, sensors, I/O, and the like. In the illustrated embodiment, the spaces include a touch sensor 924, camera 926, battery 928 and cooling system 930.
External Viewing & Audio
Typically, while the head-mounted display system is operating in the internal viewing mode, the user may have limited awareness of the outside world. For example, when the user is watching stored image based content on a head-mounted display system, the user may be unaware when someone approaches him and asks a question. As a result, the head-mounted display system may allow the user to interact with his surroundings by operating in an external viewing mode.
In the external viewing mode, a detecting mechanism (e.g., a camera on the portable electronic device such as camera 1010 of FIG. 10B or a separate camera disposed on the outside of the head-mounted device such as camera 306 of FIG. 3A) can provide for viewing outside of the head-mounted device when the user is wearing the device.
For example, the detecting mechanism can provide external image based content (e.g., images or video) of the user's surroundings which can be displayed to the user (e.g., in left image 1110 and right image 1112 of FIG. 11B). As a result, the user may see his surroundings while wearing the head-mounted display system.
The head-mounted display system can enable the external viewing mode using any suitable approach. For example, the head-mounted display system can detect when the user has selected an option to view external image based content (e.g., the user may have selected a control, issued a voice command, or made a head movement).
As another example, the head-mounted display system can enable the external viewing mode based on an indication received from the user's surroundings. For example, sensors on the head-mounted device or the portable electronic device may detect objects or sound around the user and direct the head-mounted display system to enable the external viewing mode. For instance, in response to a proximity sensor detecting an object in front of the user, the head-mounted display system can automatically activate the external viewing mode.
As another example, in response to an ambient sound detector detecting sound around the user, the head-mounted display system can automatically activate the external viewing mode. To prevent the head-mounted display system from unnecessarily switching to the external viewing mode whenever sound is detected, the activation may only occur when sound is detected above a certain threshold (e.g., 10 dB).
In some embodiments, the ambient sound detector can work together with a voice recognition module. For example, the external viewing mode may activate if words signaling a possible emergency are detected. As another example, the external viewing mode may activate if the voice of a particular person is detected (e.g., voice of a person on the user's contact list).
In some embodiments, the head-mounted display system can make adjustments in response to detecting objects or sound. For example, visual cues (e.g., arrows) can be displayed that correspond to the direction of detected sound. After seeing the cues, the user may select an option to stop the head-mounted display system from switching to the external viewing mode.
In some embodiments, the head-mounted display system can allow the user to hear sounds from the outside world without activating the external viewing mode. For example, a microphone on the exterior of the head-mounted device or the portable electronic device can provide audio through the earphones (e.g., earphones 316 of FIG. 3A) of the head-mounted device. In some embodiments, the head-mounted display system can lower the volume of the image based content that is playing so that the user can hear external sounds. For example, while the user is traveling on an airplane, the ambient sound detector may detect that a flight attendant is asking the user a question. In response to detecting the flight attendant's voice, the head-mounted display system can lower the volume of the stored image based content that is playing so that the user can hear the flight attendant's question. Alternatively, in response to detecting sound, the head-mounted display system can mute the stored image based content, and provide close captioning of the stored image based content on the display screen. As a result, the user can see what is being said in the stored image based content even if the sound is not audible. As another alternative, the head-mounted display system can automatically pause or stop the stored image based content that is currently playing so that the user can hear external sounds.
Cool Picture-in-Picture (PIP) Mode
In some embodiments, the head-mounted display system can operate in a picture-in-picture ("PIP") mode, which allows the user to see both external and stored image based content. For example, as shown in FIGS. 12A-12C, the head-mounted display system can display a PIP image frame in addition to the main image frame(s). The PIP image frame may be spaced apart and utilize a different portion of the screen than the main image frame(s) or it may be partially overlaid on top of the main image frame. Furthermore, the PIP image frame(s) may be smaller than the main viewing image frames.
Persons skilled in the art will appreciate that the PIP image frame(s) can vary in size (e.g., PIP image 1206 of FIG. 12A can vary from 800.times.600 pixels to 200.times.150 pixels). The user may adjust the size of the PIP image frame(s) by selecting an option (e.g., using controls 318 or remote control 322 of FIG. 3A).
As shown in FIG. 12A, a single PIP image frame 1206 may be placed in the top right hand corner of the display screen. It may be overlaid over a corner of main image frame 1204 (as shown).
As shown in FIG. 12B, the head-mounted display system can also place a portion of the PIP image frame in each of the main image frames (e.g., PIP image frames 1208 and PIP image frames 1210). PIP image frames 1208 and 1210 can display images in a manner that allows the PIP image frames to be viewed comfortably by the user. For example, PIP image frames 1208 and 1210 can be placed in identical locations in left image frame 1212 and right image frame 1214. As another example, PIP image frames 1208 and 1210 can be offset from each other to adjust for differences in the visual field of each eye.
As shown in FIG. 12C, the head-mounted display system can display a PIP image frame 1216 that is separate from main image frames 1218 and 1220. That is, the PIP image frame 1216 is spatially apart and in a different portion of the display screen.
Because main image frames shown in FIGS. 12A-12C can display stored image based content and the PIP image frame can display image based content from a camera, the user may continuously see activities occurring in the outside world while watching stored image based content on the head-mounted display system. It will be understood that image based content displayed in the main image frame(s) and the image based content displayed in the PIP image frame(s) can be switched. Thus, main image frame(s) can display image based content from the camera and the PIP image frame(s) can display stored image based content playing on the head-mounted device. The head-mounted display system can provide options for the user to switch between these two PIP modes.
In some embodiments, when the head-mounted display system is operating in the PIP mode, the user may select an option to cause the head-mounted display system to operate in the external viewing mode. For example, if the PIP image frame is not providing a good resolution, the user may want to enlarge the PIP image frame to full screen. In response to the user selecting an option to enlarge the PIP image frame, the head-mounted display system can display the image based content from the detecting mechanism in the left and right image frames. The head-mounted display system can then remove the PIP image frame.
Overview of Illustrative Head-Mounted Display System
Apple's patent FIG. 13 shows a simplified block diagram of an illustrative head-mounted display system 1300 in accordance with one embodiment of the invention. Head-mounted display system 1300 may include a head-mounted device 1304 that connects to a portable electronic device 1302. The system may, for example, correspond to the systems shown in FIGS. 1-12C.
Block 1314 is noteworthy being that it could include an optical subassembly 1314 configured to help optically adjust and correctly project the image based content being displayed by screen 1303 for close up viewing.
Apple credits Quin Hoellwarth as the sole inventor of patent application 20100079356, originally filed in Q3 2008.
Notice: Patently Apple presents only a brief summary of patents with associated graphic(s) for journalistic news purposes as each such patent application and/or grant is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application and/or grant should be read in its entirety for further details. For additional information on any patent reviewed here today, simply feed the individual patent number(s) noted in this report into this search engine.
