1) A method comprising:
accessing, by a processor, an audio-visual assistant that is executing on a user device, the audio-visual assistant comprising an optical detection function for detecting eye movements of a user of the device and an audio detection function that enables audio interaction by the user with the device;
determining, by the processor via the audio-visual assistant, a portion of a display screen currently being observed by the user;
identifying, by the processor, a user interface element displayed within the portion of the display screen;
receiving, by the processor, a command from the user to control the identified user interface element;
selecting, by the processor, a task to be associated with the user interface element from a plurality of tasks, the plurality of tasks corresponding to respective plurality of commands associated with the audio visual assistant; and
executing, by the processor, the selected task associated with the user interface element based at least on the received command from the user.
2) The method of claim 1, wherein the audio-visual assistant is configured to receive a first subset of the plurality of commands as input via the optical detection function.
3) The method of claim 1, wherein the audio-visual assistant is configured to receive a second subset of the plurality of commands as input via the audio detection function.
4) The method of claim 1, wherein the audio-visual assistant is configured to receive a third subset of the plurality of commands as a combination of inputs via the optical detection function and via the audio detection function.
5) The method of claim 4, wherein the command is comprised in the third subset such that the identified user interface element is selected in response to the input received via the optical detection function and the task is selected in response to the input received via the audio detection function.
6) The method of claim 4, further comprising:
converting, by the processor, the input to the audio detection function to text.
7) The method of claim 6, further comprising:
identifying, by the processor, a command in program code that maps to the text.
8) The method of claim 7, executing the selected task associated with the user interface element further comprising:
executing, by the processor, a code block associated with the command.
9) The method of claim 1, determining the portion of the display screen further comprising:
calibrating, by the processor, the user’s face such that positions of features on the user’s face relative to the display screen are mapped;
storing, by the processor, information regarding the positions in a data storage of the user device; and
displaying, by the processor, a cursor upon conclusion of the calibration on the portion of the display screen currently being observed by the user.
10) The method of claim 9, wherein identifying the user interface element further comprises:
identifying, by the processor, the user interface element displayed below the cursor on the portion of the display screen.
11) The method of claim 1, wherein the user interface element is associated with a software application being executed on the user device.
12) An apparatus comprising:
at least one processor;
a storage medium for tangibly storing thereon program logic for execution by the processor, the program logic comprising:
accessing logic, executed by the processor, for accessing an audio-visual assistant on a user device, the audio-visual assistant comprising an optical detection function for detecting eye movements of a user of the device and an audio detection function that enables audio interaction by the user with the device;
determining logic, executed by the processor for determining via the audio-visual assistant, a portion of a display screen currently being observed by the user;
identifying logic, executed by the processor, for identifying a user interface element displayed within the portion of the display screen;
receiving logic, executed by the processor, for receiving a command from the user to control the identified user interface element;
selecting logic, executed by the processor, for selecting a task to be associated with the user interface element from a plurality of tasks, the plurality of tasks corresponding to respective plurality of commands associated with the audio-visual assistant; and
logic for executing, by the processor, the selected task associated with the user interface element based at least on the received command from the user.
13) The apparatus of claim 12, the receiving logic further comprising:
visual input receiving logic, executed by the processor for receiving a first subset of the plurality of commands as input via the optical detection function.
14) The apparatus of claim 12, the receiving logic further comprising:
audio input receiving logic, executed by the processor for receiving a second subset of the plurality of commands as input via the audio detection function.
15) The apparatus of claim 12, the receiving logic further comprising:
combination input receiving logic, executed by the processor, for receiving a third subset of the plurality of commands as a combination of inputs via the optical detection function and the audio detection function.
16) The apparatus of claim 15, further comprising:
voice converting logic, executed by the processor, for converting the input received by the audio detection function to text; and
command identifying logic, executed by the processor, for identifying a command in program code that maps to the text.
17) The apparatus of claim 12, wherein the determining logic further comprises:
calibrating logic, executed by the processor, for calibrating the user’s face such that positions of features on the user’s face relative to the display screen are mapped.
18) The apparatus of claim 17, wherein the calibrating logic further comprises:
storing logic, executed by the processor, for storing information regarding the positions in a data storage of the user device; and
cursor displaying logic, executed by the processor, for displaying a cursor upon conclusion of the calibration on the portion of the display screen currently being observed by the user.
19) A computer readable storage medium, comprising instructions, which when executed by a processor cause the processor to:
access an audio-visual assistant that is executed on a user device, the audio-visual assistant comprising an optical detection function for detecting eye movements of a user of the device and an audio detection function that enables audio interaction by the user with the device;
determine via the audio-visual assistant, a portion of a display screen currently being observed by the user;
identify a user interface element displayed within the portion of the display screen;
receive a command from the user to control the identified user interface element;
select a task to be associated with the user interface element from a plurality of tasks, the plurality of tasks corresponding to respective plurality of commands associated with the audio visual assistant; and
execute the selected task associated with the user interface element based at least on the received command from the user.
20) The computer readable medium of claim 19, wherein the audio-visual assistant is configured to receive a subset of the plurality of commands as input via the optical detection function.
21) The computer readable medium of claim 19, wherein the audio-visual assistant is configured to receive a subset of the plurality of commands as input via the audio detection function.
22) The computer readable medium of claim 19, wherein the audio-visual assistant is configured to receive a subset of the plurality of commands as a combination of input via the optical detection function and input via the audio detection function.
23) The computer readable medium of claim 22, further comprising instructions that cause the processor to:
convert input received by the audio detection function to text; and
execute a code block associated with the command.
24) The computer readable medium of claim 19, further comprising instructions that cause the processor to:
calibrate the user’s face such that positions of features on the user’s face relative to the display screen are mapped;
store information regarding the positions in a data storage of the user device.
25) The computer readable medium of claim 24, further comprising instructions that cause the processor to:
display a cursor on the portion of the display screen currently being observed by the user upon conclusion of the calibration; and
identify the user interface element displayed below the cursor on the portion of the display screen.
The claims below are in addition to those above.
All refrences to claim(s) which appear below refer to the numbering after this setence.
1. A screen comprising:
a transparent panel;
a photon-generating layer in order to form each luminous pixel visible through the transparent panel, this layer being housed behind the transparent panel; and
a humanmachine interface able to control the photon-generating layer in order to modify the display of an image by the screen,
wherein the interface comprises:
a localizing device for localizing a movable permanent magnet able to be moved directly by hand by a human being in front of the panel, this device comprising for this purpose:
a network of magnetometers comprising N tri-axis magnetometers mechanically connected to one another with no degree of freedom in order to maintain a known distance between each of these magnetometers, where N is a whole number higher than or equal to five; and
an electronic processing unit able to determine the position andor orientation of the permanent magnet from the measurements of the magnetometers of this network; and
a unit able to control the photon-generating layer in order to modify the display depending on the position andor orientation determined by the device for localizing.
2. The screen as claimed in claim 1, in which the number N is higher than or equal to sixteen.
3. The screen as claimed in claim 1, in which the network of magnetometers of the localizing device is housed behind the photon-generating layer on the side opposite the panel.
4. The screen as claimed in claim 3, in which the magnetometers are placed in a given plane and distributed in this plane so as to form a plurality of columns of magnetometers that are parallel to one another.
5. The screen as claimed in claim 1, in which the magnetometers of the network of magnetometers are arranged inside a strip that encircles the external periphery of the photon-generating layer and none of the magnetometers of the network of magnetometers is placed in front of or behind the photon-generating layer in a direction perpendicular to this layer.
6. The screen as claimed in claim 1, in which the electronic unit includes a non-transitory computer-readable medium containing program instructions programmed to:
a) estimate the position of the permanent magnet relative to the network of magnetometers;
b) calculate the distance between each magnetometer and the estimated position of the permanent magnet; then
c) eliminate the N, magnetometers closest to or the Ne magnetometers furthest from this estimated position, where Ni+Ne is a positive whole number strictly lower than N;
d) acquire the measurement of the magnetic field generated or modified by the permanent magnet taken by each non-eliminated magnetometer; and
e) determine the position or orientation of the permanent magnet from the measurements alone of the non-eliminated magnetometers.
7. An assembly comprising:
a screen according to claim 1; and
a movable permanent magnet able to be localized by the localizing device.
8. A method for controlling the display of a screen equipped with a transparent panel and with a photon-generating layer in order to form each luminous pixel visible through the transparent panel, this layer being housed behind the transparent panel, wherein the method comprises:
a human being moving a movable permanent magnet directly by hand in front of the panel;
measuring the magnetic field generated or modified by the permanent magnet with a plurality of magnetometers of a network of magnetometers comprising N tri-axis magnetometers mechanically connected to one another with no degree of freedom in order to maintain a known distance between each of these magnetometers, where N is a whole number higher than or equal to five;
determining, with an electronic processing unit, the position andor the orientation of the permanent magnet from the measurements of the magnetometers of this network; and
controlling the photon-generating layer in order to modify the display depending on the position andor orientation determined by the electronic unit.
9. The method as claimed in claim 8, in which the method comprises:
a) estimating the position of the permanent magnet relative to the network of magnetometers;
b) calculating the distance between each magnetometer and the estimated position of the permanent magnet; then
c) eliminating the Ni magnetometers closest to or the Ne magnetometers furthest from this estimated position, where Ni+Ne is a positive whole number strictly lower than N;
d) measuring the magnetic field generated or modified by the permanent magnet with each non-eliminated magnetometer; and
e) determining the position andor orientation of the permanent magnet from the measurements alone of the magnetometers that have not been eliminated.
10. The method as claimed in claim 9, in which in the measuring step, the eliminated magnetometers are turned off)
11. A non-transitory data storage medium, containing program instructions for implementing a control method according to claim 8, when said instructions are executed by an electronic processor.