Yesterday the US Patent & Trademark Office published a patent application from Apple that relates to an iPhone being able to cross interact with Macs much like how Samsung's DeX works between Galaxy smartphones and Windows or Mac devices. Samsung sells DeX by stating "Open a new desktop world. Multitask between devices like a boss."  

Apple notes in their new patent filing that current methods for interacting with content displayed on electronic devices are outdated, time consuming, and inefficient. For example, some existing methods use complex and time-consuming user interfaces, which may include multiple key presses or keystrokes, and may include extraneous user interfaces. In addition, these methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-operated devices.

Apple's invention covers techniques that provide electronic devices with faster, more efficient methods and interfaces for engaging in cross device interactions. Such methods and interfaces optionally complement or replace other methods for engaging in cross device interactions.

Such methods and interfaces reduce the number, extent, and/or nature of the inputs from a user, reduce the cognitive burden on the user, and produce a more efficient human-machine interface.

For battery-operated computing devices, such methods and interfaces conserve power and increase the time between battery charges. Further, such methods and interfaces also streamline content interaction across multiple devices, which reduce unnecessary received inputs and improve user efficiency and output.

Cutting to the chase, Apple's patent focuses and dedicates the vast majority of its patent figures, at least 27 of them, to illustrate how a future MacBook will interact directly with a future iPhone via the trackpad and not via a touch screen on a MacBook.

There are four patent figures in all that illustrate how a user will be able to tap on certain keys of the MacBook keyboard to interact with an iPhone and FIG. 8K above is one specific example. Why show that if the MacBook was to implement a full screen touch display?

Apple specifically notes: FIG. 8K illustrates detecting tap gestures with contacts #7616-7619 on the keyboard of electronic device #5002 (MacBook), and in response to detecting the tap gestures while cursor #5102 is displayed on the MacBook display #5012, displaying an update to the user's grocery list.

In later patent figures Apple shows the user adding "popcorn" to the grocery store list via the MacBook's keyboard that updates the list that is associated with the iPhone that rests near the MacBook.

The user pulls up the grocery list on their iPhone and rests it close to a MacBook that will bring the iPhone UI up on the MacBook Display.

A grocery list is a simple example. Apple makes it clear that the interaction between a future MacBook and iPhone will include a wide range of applications such as a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an e-mail application, an instant messaging application, a workout support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and/or a digital video player application.

Engaging in Cross Device Interactions

It begins in patent FIG. 6A shown below where cross device interactions begin to take place. A user will take their iPhone and then hold it up to the MacBook display as shown in FIG. 6B to ensure that the back of their iPhone comes into direct contact with the MacBook display. This will turn on the interactive feature so that users will be able to turn on an app on their iPhone and be able to interact with the it on your Mac or MacBook display using a keyboard.

In patent point #191, Apple notes that in "one or more embodiments, contact sensors of at least one electronic device of the two electronic devices that are placed over each other (e.g., electronic devices 5002 and 5004) detect whether contact has been made between one electronic device and the display of the other electronic device.

In some embodiments, at least one electronic device of the two electronic devices that are placed over each other determines the signal strength of signals transmitted by the other device, and determines whether one electronic device is placed over the display of the other electronic device based on the received signal strength."

It's understandable why Apple is trying to convey the fact that the app on the iPhone is being mirrored on the MacBook by constantly showing the iPhone over the MacBook display. However it does complicate the simple fact that the iPhone App is simply being mirrored on the MacBook display. The patent visuals in some way are confusingly being conveyed.

Once the two devices are connected, the user chooses the app of interest on their iPhone that is then mirrored on the MacBook or iMac display to see things larger and make it easier to edit or create content.

That's a quick basic overview of the cross device interactions patent 20200019367. You could check out the gritty details of the patent application here.

One more Thing

In 2010 Patently Apple posted a patent report about an iMac with a touch screen display. That wasn't hinting, it was loud and clear. The news of this excited the market and Apple fans alike. It was one of those big buzz moments about a possible future Apple product. This invention was granted to Apple in 2018.

Yet even with a patent, Apple's executives damned their own patent and the idea adamantly. It was never, ever, going to happen according to them. It would create Gorilla Arm for users they explained.

The argument of course falls apart quite easily being that on an iPad Pro with a Smart Keyboard that has no trackpad, users are likely to stretch out and touch the screen for certain functions. Where are the cries of having gorilla arm now?

With that said, Apple's interesting cross device interactions patent got sidetracked by a byline yesterday about Apple hinting it isn't ruling out touchscreen MacBooks.

Out of 283 patent filing paragraphs, there's a partial paragraph with two sentences within patent point #0190 that states the following: "In some embodiments, display #5012 is also a touch-sensitive display.  In one or more of such embodiments, the user optionally performs a variety of finger inputs over display #5012 to enter user inputs via display #5012."  

Could Apple actually break down and add a full touch screen to a MacBook Pro that already has a Touch Bar? How stupid would that be? Yet under Patent FIG. 6A specifically where Apple mentions a touch display on a MacBook, we also see a Touch Bar. That's not likely to pan out unless one figure is really an iPad Pro and Smart Keyboard combination. 

Considering that the patent is about cross interaction between an iPhone and other computer devices, then technically, the one example of a touch display could in fact be a that of an iPad Pro with a Smart Keyboard interacting with an iPhone. In the general market many "2-in-1" laptops are actually Surface tablets with a keyboard.

At the end of the day, the patent is not all about a possible MacBook Pro full touch screen display but rather a way to counter Samsung's DeX as presented at the top of our report. That is what the patent is about and not the side show of a long shot MacBook Pro with a full touch display.

How embarrassing it is that so many in the media jumped on that one tiny point of a MacBook with a touch display instead of seeing the larger picture.