.. _speech_visualizer:
Speech Visualizer
=================
| The Dialog-Visualizer is a collection of tools that contains at the moment a SpeechVisualizer and a GrammarHelper.
| Their aim is to assist the interaction with the dialog.
|
.. contents::
.. _speech_hypotheses_visualizer:
Speech Hypotheses Visualizer
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Used to show results from speech recognition.
Related resources
-----------------
Component repository:`speech-visualization `_.
System startup:
- The speech recognition can be found in ``lsp-csra-supplementary.sh`` on the ``dialog`` tab.
- The component script is called ``component_speech-hyp-visualizer``
The program can be invoked with the following parameter:
-s, --scope Set the rsb-scope where you want to grab speech-results from.
-p, --speaking-scope The scope which tells if robot is speaking.
-i, --situation The scope on which situation recognition results are published.
-a, --attention The Aspect / State to listen for attention information, can be a regular expression.
-e, --addressee The Aspect / State to listen for addressee information, can be a regular expression.
-d, --debug debug-mode, shows more information.
Related projects:
- :ref:`dialog_manager`
- :ref:`speech_recognition`
- :ref:`speech_recognition_tool`
Interfaces
-----------------
Currently there are two interaction islands, the kitchen and entrance.
=============================================================== ========================================
Scope (Listener) Type
=============================================================== ========================================
``/speechhypotheses`` `SpeechHypotheses`_ , `DialogAct`_
``/situation/result`` `ClassificationResultMap`_
``/saytask/SAY`` `TaskState`_
``/flobi/speakingstate`` String
``/speechhypotheses/visualization/debugmode`` String
=============================================================== ========================================
.. _DialogAct: http://docs.cor-lab.de//rst-manual/trunk/html/generated/sandbox/package-rst-dialog.html#message-dialogact
.. _SpeechHypotheses: http://docs.cor-lab.de//rst-manual/trunk/html/generated/sandbox/package-rst-dialog.html#rst.dialog.SpeechHypotheses
.. _TaskState: http://docs.cor-lab.de//rst-manual/trunk/html/generated/stable/package-rst-communicationpatterns.html#rst.communicationpatterns.TaskState
.. _ClassificationResultMap: http://docs.cor-lab.de//rst-manual/trunk/html/generated/sandbox/package-rst-classification.html#rst.classification.ClassificationResultMap
Debug Mode
-----------
Debugmode can be enabled and disabled via cmdline-properties or an rsb-event.
The scope of the debugmode-listener is always ``speech-hypotheses-scope + /visualization/debugmode``.
The rsb-event to handle debugmode must contain a string-message ``ON`` or ``OFF``, not case-sensitive.
Enabling debugmode entails:
* a textFrame for grammarTrees shows up.
* grammarTrees (if available) are shown.
* type of incoming dialogact is printed to terminal.
Examples
--------------
- start some speech-recognition-software, that sends its sprechhypothese or dialogacts via rsb.
- start visualizer with scope to listen to ``./speechvisualizer -s /result_scope/``
If you give no scope, default will be "/citec/csra/home/hallway/entrance/speechhypotheses".
- see what recognizer sends as result.
A small bar at the bottom of the screen appears. The results from the speech-recognition will be posted there.
The current hypothesis is shown in the first line, below is the history of last hypotheses.
.. figure:: /images/software/speech-hyp-visualizer/speech-visualizer-text-final.png
:figclass: align-center
If the result is not final yet, the text will be greyed out. If it is final, the text is black. Filler-words will only be shown as long as they are not final.
.. figure:: /images/software/speech-hyp-visualizer/speech-visualizer-text-not-final.png
:figclass: align-center
You can also let the visualizer show the speaking and attention / listening state of the robot-system.
Therefore you can invoke it with the command -p, which tells the visualizer to show if the robot speaks on a certain scope.
There is a small section at the right end which becomes colored if the system is speaking.
.. figure:: /images/software/speech-hyp-visualizer/speech-visualizer-system-speaking.png
:figclass: align-center
Also, you can invoke it with the parameters -a and -d to to show the attention and addressee state of the system.
When the backgroundcolour of the textbox is blue, the system has attention to something.
.. figure:: /images/software/speech-hyp-visualizer/speech-visualizer-has-attention.png
:figclass: align-center
When the backgroundcolour of the textbox is green, the system has attention and an addressee.
.. figure:: /images/software/speech-hyp-visualizer/speech-visualizer-has-addressee.png
:figclass: align-center
If you enable debug-mode, you will see a second frame above the recognition-bar. It will show grammar-trees from incoming `SpeechHypotheses`_ or `DialogAct`_.
.. figure:: /images/software/speech-hyp-visualizer/speech-visualizer-grammar-tree-frame.png
:figclass: align-center
.. _grammar_helper:
Grammar Helper
^^^^^^^^^^^^^^
Used to show possible speech commands for an interaction with the dialogsystem.
Related resources
-----------------
Component repository:`speech-visualization `_.
System startup:
The start script grammar-helper.sh can be found at ``/opt/speech-visualizer/bin/grammar-helper.sh``.
The program can be invoked with the following parameter:
-r, --scp-scope Set the rsb-scope where you want to call the :ref:`speech_command_proposer`.
-d, --debug debug-mode, shows more information.
-l, --dialog-act Set the scope where dialogacts should be published.
Related projects:
- :ref:`speech_visualizer`
- :ref:`speech_command_proposer`
Interfaces
----------
The property of ``-r`` or ``--scp-scope`` serves as prefix.
=============================================================== ========================================
Scope (Listener) Type
=============================================================== ========================================
``prefix/grammarhelper/debugmode`` String
``prefix/grammarhelper/rulechange`` String
=============================================================== ========================================
The scope for rulechange is used to inform the grammarhelper that active rules have changed.
It is used by the :ref:`speech_command_proposer`.
Debug Mode
----------
Debugmode can be enabled and disabled via cmdline-properties or an rsb-event.
The scope of the debugmode-listener is always ``scp-scope + /grammarhelper/debugmode``.
The rsb-event to handle debugmode must contain a string-message ``ON`` or ``OFF``, not case-sensitive.
Enabling debugmode entails:
* the grammar-helper frame will be switched to debugmode.
* grammar control section lets you set and ask for the jsgf-grammar the :ref:`speech_command_proposer` uses.
* two ``getRules`` buttons to grab rule-lists from the :ref:`speech_command_proposer`.
Examples and Screenshots
-------------------------
| The appearance of the grammar-helper differs for the debug mode and no-debug mode.
| Below the two frames are shown for comparison. Only one mode can be active at a time.
|
.. figure:: /images/software/grammarhelper/overview.png
:figclass: align-center
Basic usage
***********
| Start the :ref:`speech_command_proposer` .
| For example ``./speech-command-proposer -s /ScopeToMyServer/ -g /path/to/my/grammar.gram``
| Start grammar-helper with scope to make remote-calls ``./grammar-helper -r /ScopeToMyServer/``
| A frame at the right of the screen appears.
| After the tool has started, it will try to grab a list of active rules from the :ref:`speech_command_proposer`.
| It will show an example-sentence for every rule in a list in the white list-box.
| The rule is updated autonomous whenever the list of active rules or the grammar is changed.
| The two checkboxes at the top-right enable their entitled action on click when checked.
| So you can retrieve new examples or send a dialogAct for the clicked entry.
|
Example for debugmode
*********************
| In debugmode you have more control about the grammar and what rules are shown.
| To set a grammar to make suggestions from, simply type in the file-path in the top text-field and press setGrammar.
| The label beneath the button will tell if setting the grammar worked fine.
|
| If you want to check which grammar the :ref:`speech_command_proposer` is using, simply click checkGrammar.
| In this case, the text-field beneath the button will be filled with the grammar path.
|
.. figure:: /images/software/grammarhelper/grammar.png
:figclass: align-center
| If your are content with the used grammar file, you can retrieve its rules by hitting the getAllRules or getActiveRules button.
| It will show them up in a list in the white box below.
| Active rules are autonomous updated whenever you change the grammarfile.
| Now you can click on an entry in the list to get an example sentence for this rule (shown in the white box at the bottom).
| Also there is a checkbox above the rule-list which will allow you to send dialogActs for the clicked list-entry.
|
.. figure:: /images/software/grammarhelper/debug.png
:figclass: align-center
.. note:: Active rules are handled by the :ref:`speech_command_proposer`!
.. note:: The scope for sending dialogActs is passed through to the speech-command-proposer. The scp will also send the dialogActs.