Activities

No upcoming events

Intelligent Systems Lab Project: HWGUIDE

Participants

• Simon Jellinghaus
• Sebastian Meyer zu Borgsen
• Christian Wall

Project Supervisors

• Stefan Rüther

Motivation

The tasks in sheltered workshops are often monotonous since the worker's disabilities impair the complexity of performable tasks. An assistance system could increase the complexity of tasks by guiding the worker through more difficult process steps. Additionally an assistance system could provide a quality control of the work. The system could notice the error rates of every worker. Workers with low error rates can be applied more difficult tasks. Since the worker improves in performing the tasks over time, the system ideally adapts to the growing user's skill. The efficiency of the whole work process is enhanced by the system. The sheltered workshop is able to serves commercial and private customers safer and can thus ensure a corresponding revenue.

Application Szenario

A scenario is shown in the following video. After the worker logs in the system, the Tabletop-touchscreen shows the placement opportunities for five boxes. Each box contains many different items. After a box is identified by the RFID-Reader, the the system recognizes the position where the box is put on the touchscreen and saves the position. When all boxes are placed, the worker has to scan and placed the packet, which he has to fill. The beamer projects the number of contained items (green background) and the needed items (blue background) for the actual packet on the top of every box. When all pieces which needed are in the packet, the user get a visual feedback for this box (the background of the Tabletop-touchscreen becomes green).

Objectives

The goal of the HWGUIDE - Project was to develop an interactive assistance system to help disabled workers to pack a packet which includes up to five different pieces. For the development of a handicapped accessible order picking assistants we started from an existing project that already integrated some hardware and technologies. The existing platform was delevoped by the supervisor (Stefan Rüther). The system features: a projector, a kinect, a hand tracking system, BPMN business processes, RFID, augmented reality, barcode, etc.. The focus of the project is to implement a barrier free graphical user interface, which accounts the disabilities and the abilities of the worker. Also the assistance system must have an quality control of the work to minimize the supervision and to increase customer satisfaction.
Another goal was to utilize BPMN (Business Process Model and Notation) to control the different workflows and communication within the sheltered workshops.

Description

HWGUIDE was designed as a toolkit for sheltered workshops. This toolkit is designed to be fully modular with the model, view, control pattern. In this way the gui observs the model and displays all changes. The controller handles interaction and manipulates the model. The project contains a full model of the table contents as internal representation like the number of boxes, content of the boxes and so on. It uses the Activiti-framework with business processes as the controller of the system. This processes are used to describe the workflow in sheltered workshops. A business process which describes the packaging process was build as an example and prove for the abilities for the HWGUIDE toolkit. Through the Activiti-framework the task master is able to change workflows are create entirely new ones without programming. We designed an easy to use graphical user interface which is adjusted to the needs of handicapped workers. The GUI is based on the java processing library. This library makes in really easy to improve the design. In order to match the personal qualification of each worker, we wanted to be able to individually determine the number of boxes to place. We implemented an automatic layout algorithm that renders a positioning of the set for boxes for a given number. dSensingNI was used to track the hands of the worker. To be able to resolve the interaction with objects on the table, a subsystem for Activiti had to be implemented as classificator. These classifcator contains some simple intersection tests to test whether the hand reaches into the interaction region of any object on the table. In addition these predictions are stabilized over multiple frames from dSensingNI to eliminate noise.

Integrated hardware:

• Touch screen
• Tabletop-touchscreen
• RFID, Barcode scanner
• Projector for the tabletop projection
• Kinect calibrated for the detection

The construction:

How to start the HWGUIDE-system:

Before you can use the HWGUIDE-system you have to start the windows computers from the given construction, the beamer and the kinect with related software dSensingNI. To start our HWGUIDE-system you need to unzip the file " HWGUIDE-Software.zip", that is stored in our project folder. Then you must open the folder to launch the file "isy.bat".

The file executes four software components:
• spread (The Spread toolkit provides a high performance messaging service that is resilient to faults across local and wide area networks)
• activiti (It is a light-weight workflow and Business Process Management (BPM) Platform targeted at business people, developers and system admins.)
• RFID-reader (use RSB)
• HWGContol (include the actual programming code)

Now the worker can log in and begin to work!

Results

The toolkit contains a fully working demonstration for commissioning a package. Tracking of hands works really well. The tracking of boxes with dSensingNI does not work so well. It is able to interact with the toolkit but the framework has its difficulties to track the changes to the environment. The classificator showed in some evaluations to be able to predict the interaction in most cases successful which makes running the demonstration possible without tracking the boxes exactly.

Discussion and Conclusion

The System could guide a disabled worker, so that he easily is able to pack a package, although he can not count. It would be no Problem when the combination of pieces for a box would varied. A Problem could be, that the worker has to pick up only one piece. Our system can not recognize if the worker is picking up two or more pieces. In some cases it could be very troublesome to grab extra for every piece in the box.

Outlook

To make the system capable for real-world application, the system has to be connected to a database which provide the system with the various jobs for the worker. Also the datas from the worker (like the error rate ode skill level) has to be stored and archived. To incorporate different skill levels in the system the number of boxes (there are currently 5) has to vary. So for example for easy jobs the worker gets only two different boxes. To realize that the system recognizes how much pieces the worker have taken there could be used a scale in the worker table to calculate from the weight change of the box how many pieces are in the workers hand.

Literature

Towards Optimal Worker Assistance:
• A Framework for Adaptive Selection and Presentation of Assembly Instructions
  A. Bannat, F. Wallhoff, G. Rigoll, F. Friesdorf, H. Bubb, S. Stork, H. J. Müller, A. Schubö, M. Wiesbeck, M. F. Zäh


• Optimizing Human-Machine Interaction in Manual Assembly
  Sonja Stork, Christian Stößel, and Anna Schubö
  
• dSensingNI - A Framework for Advanced Tangible
  Interaction using a Depth Camera Florian Klompmaker