Emotions-aware digital twins for manufacturing

(1)

ScienceDirect

Procedia Manufacturing 51 (2020) 605–612

This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of the scientific committee of the FAIM 2021.

10.1016/j.promfg.2020.10.085

This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of the scientific committee of the FAIM 2021.

www.elsevier.com/locate/procedia

30th International Conference on Flexible Automation and Intelligent Manufacturing (FAIM2021) 15-18 June 2021, Athens, Greece.

Emotions-aware Digital Twins For Manufacturing

Anna Florea

^a,∗

, Andrei Lobov

^b

, Minna Lanz

^a

aTampere University, Kalevantie 4, Tampere 33100, Finland bNorwegian University of Science and Technology, Trondheim 7034, Norway

Abstract

Digital manufacturing employs different enabling technologies to represent various aspects of product and its manufacturing processes to be able proactively make better, i.e. informative decisions. Digital twins is the recently developed approach to have a representation of a product and/or a manufacturing site in digital space allowing to give feedback and predict development of the product or the system. As human remains widely involved in most aspects of manufacturing, an adequate digital representation of a human contributes to completeness of a digital twin.

This article proposes architecture for emotions-aware digital twins. Such architecture would allow human to keep the locus of control by better comprehending, communicating relevant data and promoting situational awareness. This work can contribute to reduction of hazardous situations at the production floor, increase products quality, and facilitate innovation and creativity in design tasks.

This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of the scientific committee of the FAIM 2020.

Keywords: digital manufacturing; emotion modelling; situational awareness; product development; digital twin; affective computing.

1. Introduction

In order to adapt to changing economy and stay competi- tive, modern manufacturing enterprises undergo organisational and technological transformation leading to lean, flexible, and strategically aligned entities widely employing state of the art technology to realize its operations and implement manufacturing processes.

This transformation process is often discussed through the prism of several complementing concepts: Industrie 4.0 (I4.0), Smart Manufacturing and Digital Manufacturing. I4.0 provides a reference architecture model, that brings together three dimensions across which information must be flowing for a manufacturing system to satisfy needs of modern economy[1]. Con- cept of Smart manufacturing concerns with holistic view on a product life cycle and ways to automate this approach allowing learning and further extension of the manufacturing system[2].

Digital manufacturing in its turn focuses on technologies for information management through the product life cycle and meth- ods to assist decision-making in this context[3].

∗Corresponding author. Tel.:+358-294-52-11 E-mail address:anna.florea@tuni.fi (Anna Florea ).

Digital manufacturing employs different enabling technologies to represent various aspects of product and its manufacturing processes to be able proactively make better, i.e. informative decisions. Digital manufacturing concerns with information- sharing models, collaborative design tools, simulation tools, and computer integrated systems to allow design, analysis and redesign of product, processes and factory.

Digital twins (DT) is the recently developed approach to have a representation of a product, a manufacturing asset, or a manufacturing site in digital space allowing to give feedback and predict development of the product or the system. High- fidelity of representation[1] is often mentioned among critical properties of a DT, together with the ability to communicate with its physical counterpart[4]. Three tests are suggested in [5] in order to evaluate virtual system’s accuracy with respect to visual representation, performance and representation at later stages of life cycle.

While visual representations improve naturally driven by progress in visualisation technologies, progress in the remain- ing two dimensions requires more deliberate effort. However, often connection to the physical product is lost once it abandons manufacturing site, therefore most of the progress is possible in performance accuracy of the DT. Performance test requires that behaviour of the DT entity matches as close as possible the

30th International Conference on Flexible Automation and Intelligent Manufacturing (FAIM2021) 15-18 June 2021, Athens, Greece.

Emotions-aware Digital Twins For Manufacturing

Anna Florea

^a,∗

, Andrei Lobov

^b

, Minna Lanz

^a

Abstract

1. Introduction

30th International Conference on Flexible Automation and Intelligent Manufacturing (FAIM2021) 15-18 June 2021, Athens, Greece.

Emotions-aware Digital Twins For Manufacturing

Anna Florea

^a,∗

, Andrei Lobov

^b

, Minna Lanz

^a

Abstract

1. Introduction

(2)

behaviour of the physical system. This also means that operational conditions of the physical system are known to the DT and it can adjust own performance to respond to those.

The three tests originally refer to DTs of such product as complex engineering system, where physical conditions are of biggest interest in performance representations. In context of manufacturing applications there are also other factors worth considering during operation of manufacturing assets as well as the entire factory: since human remains widely involved in most aspects of manufacturing, an adequate digital representation of a human increases accuracy of DT performance. Wearable devices and advanced media processing allow capturing a variety of parameters related to human activities, which have effect on system performance and safety. Leveraging these technologies and advances in affective computing, it becomes possible to also bring emotional state of a human to the mix, therefore extending representation of human beyond set of parameters usually shared across humans and manufacturing assets.

This article proposes an architecture for Emotions-aware Digital Twins(EADT). Such architecture would increase accuracy of DT performance and allow human to keep the locus of control by better comprehending, communicating relevant data and promoting situational awareness. This work can contribute to reduction of hazardous situations at the production floor, increase products quality, and facilitate innovation and creativity in design tasks.

The rest of the article is structured as follows: Section2de- scribes approaches to emotion modelling in software systems and provides motivation for introducing emotion awareness capability to digital twins; Section3describes use-cases where manufacturing applications and enterprises would benefit from implementing emotion-awareness in digital twins; Section4de- scribes high level architecture for emotion awareness and dis- cusses how its components align with reference model of digital twin. Conclusions and considerations for future work provided in Section5finalize the article.

2. Background

While manufacturing industry undergoes transformation leading to higher degree of digitisation on every level, human remains important contributor, being the main resource for information processing, troubleshooting, and decision making, especially in unconventional situations. Therefore, it becomes important that manufacturing systems helps to grow, maintain and develop these skills to adjust to the newly emerging working conditions. This can be accomplished by implementation of human-centered applications. In this context ability to capture, understand and communicate emotions becomes particu- larly important. Digital twin, as one of prominent components of digital manufacturing is seen as convenient tool to implement emotion awareness in next generation of manufacturing systems.This section provides an overview of approaches to emotion modelling for software systems and provides motivation for introducing emotion awareness capability to digital twins for manufacturing.

2.1. Emotion modelling

For technical system to be able to capture, understand and communicate emotions, an adequate emotion representation must exist. This sub-section gives an introduction to emotion modelling by providing a brief overview of existing emotion theories and approaches to emotion modelling in software systems.

2.1.1. Emotion theories

Numerous theories were introduced over time in attempt to explain process of emotion formation:

• Neurologicalapproach sees emotion as a result of process occurring solely within the brain([6,7]).

• Conditioned responsetheory, also referred asphysiologi- calin [8] defines emotion as a reaction to the physiological reaction of the body to an event (e.g. raised adrenaline level will cause the emotion of fear).

• According toappraisaltheory emotion forms under influence of one’s expectations from an event and event evaluation([9,7,8]).

• Thayer’s emotion modelcharacterises emotion through two dimensions:arousal(i.e. energy or intensity of emo- tion) ranging from low to high, andvalenceof the emotion, which can be positive or negative([10]).

Because of diversity of the views towards emotion origin and structure, there exist multiple scales for emotion classifications.

Some research distinguishes as little as two basic emotions (i.e.

pain and pleasure) [11], while other researchers introduce more elaborate lists, and consider more orders of emotions. [6] men- tions fear, anger,sadness, andjoyas most commonly listed.

As a result these emotions are often present within list of emotions selected for modeling [9,8]. Another popular approach to selection of emotion list for modeling (e.g. [12,7]) is to follow the emotions as reflected through universal facial expression introduced by Ekman [13], i.e.disgust,sadness,happiness,fear, anger, andsurprise.

2.1.2. Emotion modelling in software systems

Ability to capture, understand and communicate emotions is of particular interest in human-centered applications and related technologies. Among many reasons for such interest are the facts that emotions are an integral part of decision-making along rational thought, while arise 3000 faster [6,14].

Term’affective computing’coined by Rosalind Picard [6]

refers to”computing that ”relates to, arises from, or influences emotion”. Four categories of affective computing are defined, based on computer’s ability to recognise and express affect.

Field of synthetic emotions[7] researches ways to synthesise emotions in software agents. Main applications areas are gaming [12], entertainment, teaching and e-learning[15], and healthcare. A variety of approaches to formal representation of emotions exist. Particular design is mostly influenced by target application and chosen underlying emotion theory.

(3)

Domain of e-learning is broadly evaluated in [15]. The research aims for understanding of effect of learners’ feelings on motivation, self-regulation and academic achievement. It studies role of emotions in online and blended learning experience, in order to understand importance of information about learners emotions for learning analytic applications. Study structured using the framework of Community of Enquiry, which consid- ers cognitive presence, social presence, teaching presence, and emotional presence. A method to synthesise emotions in order to allow affective robotics system based on Fuzzy Grey Cog- nitive Maps and Thayer’s emotion model is proposed in [10].

Process of emotion representation through procedurally generated auditory icons is described in [12], aiming for interactive application, such as games, social robotics, animated animals and other interactive objects. An approach based on proto emotions is proposed in [11] with suggested potential application for ambient intelligence.

Some approaches attempt to reflect the fact, that emotion forms faster, then rational thought. For example emotion formation is defined as a twofold process in [9]. It distinguishes between two types of emotion:cognitive, andreflexemotions. The approach is used to implement emotions in a synthetic agent - software intended to imitate human intelligence. Agent processes input generated by external environment in two parallel routes: cognitive and reflex. Result that becomes available first is set to be the output emotion.

Another important aspect of emotion modelling is presence of memory about past emotional states or past inputs. Approach proposed in [16] builds on sensor-effector agent architecture, and relevance of an emotion is calculated using polynomial ex- trapolation. Multilayered cellular automata is used to represent sensor memory. Concept of echoic memory introduced in [11]

as ability to store previous emotion/state for a short period of time. It then contributes to development of stronger reaction if same activation signal persists.

Typically applications consider four to six discrete emotions.

However there exist approaches, that only use two. For example, [11] takes a bottom-up approach to intelligence, seeing it as emerging property of a system. Emotions are seen as alarm system activating various responses. The two basic emotions(called proto emotions) are: pain and pleasure.

As it was mentioned above, some applications introduce memory as additional context parameter for emotion formation.

Another approach to put emotion into context is by considering personality of the agent. [8] combines cognitive event appraisal with emotion elicitation conditions to derive final emotion.

Stimuli is evaluated against agent’s goals and beliefs in its current state of affection. Implementation of the approach employs Fuzzy logic. Representation of emotional state change proposed in [6] bases on Hidden Markov Model. In this approach transitions between states may occur with different probabilities and model can be trained for particular individuals, meaning set of probabilities will reflect emotional profile of particular individual.

Temporal dimension also addressed in different ways. Some researchers distinguish between long lasting states, referring to them as moods, and states lasting for shorter periods of time,

i.e. emotions[15]. Also, while many applications target past and present, there is effort to develop tools for emotional state prediction. For example [14] applies dynamic continuous factor graph model to predict emotions, the discovered patterns are modelled as factor functions and user emotional changes over time are modeled as Markov chain. Studied factors are grouped into three categories: attributes, temporal, social, and include, for example location, friend’s mood, past states, etc.

Type of information required for emotion modelling depends on the purpose of the application. Applications concerned with emotion synthesis primarily use information about the environment, trying to identify relevant events. Emerging emotion is then synthesised based the event and selected emotion emer- gence mechanism. Applications, concentrating on understanding emotional state of humans leverage inputs that represent emotion symptoms.

Manifestation of emotions takes different shapes. A classic example of emotion manifestation is facial expression [13](i.e.

Eckman faces). Being integral part of a human emotions also affect physiological parameters such as heart rate, body tempera- ture, blood pressure, pupil dilation, etc[6]. Emotional state may also be revealed through voice modulation, manifest through written text, posture, etc. Modern technology allows capturing most of the manifestations. However emotion inference is a complex task for a series of reasons: for example, emotion manifestation can be hidden or modified by an individual.

2.2. Digital twins for manufacturing and emotion awareness The concept of digital twin(DT) was initially introduced to assist in design, manufacturing and maintenance of complex engineering systems and applied in context of aerospace industry [17]. In manufacturing industry this approach is used to have a representation of a product, a manufacturing asset, or a manufacturing site in digital space allowing to give feedback and predict development of the product or the system. High accuracy of representation[1] is often mentioned among critical properties of a DT, together with the ability to communicate with its physical system it mirrors[4].

In context of manufacturing, the concept of DT is often mentioned in close relation with simulation and visualisation, where main purpose of the DT is employ virtual models of physical objects for simulating their behaviour [18]. DT, according to [18], is expected to fulfill following functions: virtual verifica- tion, simulation execution, ultra-high-fidelity real-time monitoring, prediction and diagnostics, process optimisation and improvement.

Scope of DT may vary by the type of physical entity or system it represents, phase of digital manufacturing it serves, or life cycle phase of the physical twin. Three categories of DTs based on the system they represent were identified in [4], i.e.

manufacturing asset,factory,people. Following phases of dig- ital manufacturing, DT may be tailored to better serve either of phases: design, production or control [3]. Research in product life cycle management emphasises importance of DT evolution along the product life cycle [17], similarly factory scale DTs

(4)

are expected to evolve along with the manufacturing enterprise commencement and operations [19].

Main resources of a manufacturing enterprise can be summarised as: human, machine, materials, and environment [20].

Currently available DTs for manufacturing mainly target machines and other manufacturing assets, and factories(i.e. combination of the resources) [4,19]. Since accuracy of the virtual representation is a critical property of DTs, various implementations must constantly seek ways to improve it.

As it was mentioned in Section1, accuracy of visual representations improve naturally driven by progress in visualisation technologies, while improvements in performance accuracy and maintaining connection to physical system throughout life cycle is more challenging task. Since often connection to the physical product is lost once it abandons manufacturing site, most of the progress is possible in performance accuracy of the DT. Perfor- mance test, suggested to evaluate accuracy of the virtual model [5], requires that behaviour of the DT entity matches as close as possible the behaviour of the physical system. This also means that operational conditions of the physical system are known to the DT and it can adjust own performance to respond to those.

Additionally to physical operation parameters, there are other important factors to be captured as operational conditions in manufacturing applications. One of such factors is human.

Since human remains widely involved in most aspects of manufacturing, an adequate digital representation of a human increases accuracy of DT performance. Beyond that, since growing complexity of manufacturing systems increases demand in highly skilled professionals, new generation of manufacturing systems must be built in human-centred manner, in order to support and sustain development of new skill profile. Soft skills, together with creative thinking are given big importance in this context [21]. Therefore, emotion awareness is an important feature to implement in modern manufacturing applications.

Since DTs can only be implemented for systems capable of sensing its operational context[17], often employ advanced solutions for big data processing[18, 4], and are expected to communicate results of own execution not only to their physical twin, but also other information systems, and essentially intended to construct cyberspace for smart factory [1], DT becomes most prominent point in modern manufacturing systems to implement emotion awareness.

Table 1. Use-cases for EADT Use-Case Descriprion

UC1 Emotion awareness for improved information delivery UC2 Emotion awareness for ergonomics

UC3 Emotion awareness for professional development UC4 Emotion awareness at enterprise scale.

Moreover, being one of the core tools of digital manufacturing, DTs naturally employ technologies and tools that are seen as main pillars of future manufacturing systems, i.e. cloud computing, Internet of things, cyber-physical systems, and big data. As consequence, same technologies become effortlessly available for implementation of emotion awareness capability,

enabling access to input data, tools for information processing, communication and storage.

Following section introduces four use-cases for Emotions- aware Digital Twins(EADT) and explains how they fulfill industry needs and requirements of manufacturing enterprises of the future.

3. Use-cases for EADT

Emotion-awareness capability allows DTs responding better to industry needs and requirements of manufacturing enterprises of the future, contributing to improved information delivery, ergonomics, professional development and other application at enterprise scale.

Existing DT applications for manufacturing can be grouped into three categories[4]: DTs of manufacturing assets, factory scale DTs, and DTs of people (i.e. employees). The property of emotions-awareness within the scope of present research relates to the DTs falling under the first two categories: DTs for manufacturing resources and factories. This section introduces four use-cases for Emotions-aware Digital Twins (EADT) contributing to implementation of smart manufacturing system as summarised in table 2.2.

3.1. Emotion awareness for improved information delivery Emotional state affects speed of perception and decision making, and impacts cognitive processing very early in the process[22]. Being an integral part of decision-making process it also affects the actual content decisions made[23]. There are numerous tasks and activities in a manufacturing site, where professionals are expected to decide on their next steps based on the supplied information.Typical situation, where emotion awareness could assist in information processing could be assessment of monitoring data, and compliance with safety procedures.

Emotion awareness can help improved delivery of visual information. Content can be delivered to the end-user in a format, that is adjusted to user’s emotional state. Studies show, that emotional state affects perception of brightness [24] and speed of word recognition, where words directly correlating with the emotional state are recognised faster[22]. Some examples of ad- justable presentation parameters include font size, layout, color scheme, presence of animations and their pace, sound effects and visual effects, or even vocabulary used in verbal constructs of the interface. Objective of the adjustment may be enhanced perception, faster decision-making, etc.

Implementation of safety measures can benefit from emotion-awareness as well. While modern tools are being applied to ensure efficient safety training[25], with emotion aware systems it would be possible to notice potential risks of break- ing safety rules. Those events may not necessary occur inten- tionally, but caused by ones emotional state, leading to low attention, reduced field of view, reaction time, increased response to distractions, etc. For example, studies show that angry and happy people would estimate risks being lower, and will be

(5)

more confident about own judgment compared to people ex- periencing sadness [26].

3.2. Emotion awareness for ergonomics

There are numerous macro-factors defining implementation of ergonomic working environment. However, in individual cases emotional state may introduce disturbance, that will re- duce effect of certain measures. Since modern systems com- posing ergonomic environment are meant to adapt individual needs, information about emotional state can serve as input for customisation. Two prominent applications are the field of man- machine coordination and adjustment of ambient conditions.

Smooth coordination between operators and machines is highly important. Emotional state influences reaction time, attention, and speed of decision making [23]. A machine, aware of operator’s state may adjust its operation parameters (especially cobots and CNCs), and in advanced cases also adjust own interface, both visual and haptic. Studies also show importance of concordance between emotional state and posture in solving tasks requiring creative thinking [27].

There is a two-way relationship between ambient conditions and emotional state: emotional state affects perception of environmental conditions, however, ambient conditions may affect emotional state [24]. A basic application would include adjust- ing local conditions at a work station in order to respond to operators emotional state. For example, an ability to suggest a better pose at the work station or simply a reminder to take a break can affect a well-being of the worker.

3.3. Emotion awareness for professional development Although systems are getting smarter, and amount of auto- mated processes and procedures grows continuously, humans still remain main resource for information processing, troubleshooting, and decision making. Growing complexity of manufacturing systems leads to increasing demand in highly skilled professionals[21]. Combined with shortage of skilled work- force in some industries due to massive retirements, this trend calls for efficient professional training solutions.

Professional training and knowledge sharing become of greater importance in modern enterprises. Training is delivered in a variety of formats and heavily involves technology.

It is known that for efficient learning process it is important to maintain lower levels of frustration from failure and mis- takes in order to sustain motivation and so-called self-propelled learning[6]. In this context, emotion awareness becomes a key feature of a virtual training solution. Emotion awareness could help to spot boredom and frustration, help providing tailored feedback to maintain motivation.

Additionally to cognitive tasks mentioned in the beginning of the section, humans are seen as a key source of innovation and creativity in manufacturing enterprises of the future. It is recognised, by the manufacturing enterprises, that organisational measures must be taken to utilize this potential. Workload management to enable innovation activities seen as important step for transformation to happen [3]. Emotion awareness could

help implementing the process by learning working patterns of employees and teams and deliver customised conditions to different people involved in innovation activities , and help detect- ing any innovation blocks. There is correlation between emotional state and internal perception of time [28], that could be used to sustain the feeling of ’flow’. Another work proposes an approach, where knowledge about personality traits and prefer- ences is used to distribute assembly tasks between employees [29].

3.4. Emotion awareness at the factory scale

Emotion awareness implemented for factory-scale DT may assist manufacturing enterprise in many ways, for example developing better communication practices and maintaining employee engagement [30]. Enterprise-scale applications will have richer and more diverse set of information sources about individual’s emotional state, and greater computational capac- ity, which enables more accurate implementations.

Emotion awareness implemented at the factory scale DTs would allow to bring more insight into incident investigation process. An availability of emotion profile of the involved par- ties could help better understanding of the situation and develop improved processes, taking into account knowledge on tendencies in risk assessment[26] and relevance of communication within existing emotional context [23].

From the organisational perspective it could assist transition towards digital manufacturing by providing feedback on accep- tance of changes and designing strategies tailored for particular site. Capability of emotion-awareness would provide additional inputs for the change planning and it can facilitate sentiment analysis when a new process is introduced. It will also support design of appropriate communication much needed throughout transition[22,26].

4. Architecture

Use-cases for EADT described in previous section cover a rather diverse set of applications. As consequence, specific requirements may differ for particular implementation. For example, some implementations (e.g. cases for improved information delivery) may require responsive real-time behaviour, while enterprise-scale applications may afford running complex off-line analytics for solving accurately more complex issues.

In similar manner some applications would bring more benefits when tailored for particular person and require thorough design solutions concerning privacy, while for others anonymous data may suffice. Information flow in DTs must be two-way by defi- nition [4], even if some applications may only require upstream link, both links would already exist in the system.

Taking into account the above-mentioned nature of the requirements, first a high level architecture is proposed to cover main functions an EADT must implement. Then each function is discussed in more detail, and its inputs, outputs and intended behaviour is described. Finally functions are aligned with DT reference model proposed in [4].

(6)

4.1. High level architecture for Emotion awareness

Figure1contains a high level overview of the approach to emotion awareness implementation in a DT environment. A system must implement three behaviours in order to allow emotion awareness. Those behaviours are:perception,interpreta- tion, andaction. Execution of each behaviour must comply with privacy and security policies set within the system.

Fig. 1. Reference architecture for emotion aware digital twin environment

Following sub-sections provide detail on inputs, scope, and outcomes of execution for each function as portrayed in figure 2.

4.1.1. Perception

Emotional state manifests in a variety of ways. It can be captured through observation of behaviour, physiological parameters, and content produced by a human. Those numerous parameters can be captured in different formats, e.g. numeric format, media data (i.e. video, audio, images), and text. Emotional state can be inferred based on a single input, or a combination of inputs from different sources.

As presented in figure 2perception function is accomplished by Emotion Inference Module (EIM). It processes input data on emotion manifestation, and formulates an concludes o emotional state of the user interacting with the system.

EIM consists of two components: Emotion model and inference engine.Emotion modellies at the core of the module, it may vary from one DT to another. Emotion model is intended to represents set or spectrum of emotions, their parameters, and, if necessary, dynamics (i.e. transition from one emotional state to another). As outlined in Section2.1, there is a variety of ways to represent emotions. Particular choice of modelling approach will be influenced by objectives of the EADT and availability and type of input data.Inference engineis responsible for making conclusion about the captured emotional state. The conclusion is formulated in terms of emotion model, yet no assessment of perceived emotion with respect to system performance objective is done. Perceived emotionEis then passed on to the interpretation module.

4.1.2. Interpretation

Once system is able to identify emotional state of the subject, next step is to evaluate this emotion with respect to the purpose of the system. The system must evaluate detected emotion against the reference model for purpose of building complete context of its operation. This function is implemented by Emotion Interpretation Module(EIM).

Emotion Evaluation Moduleassesses perceived emotion us- ingsystem performance profileand relevant live data about the system. System Performance Profile feeds evaluation module with information on performance objectives and retrospective of performance aligned with emotional states. The outcome of this assessment isE, emotion accompanied with a conclusion of its effect on the system. Again, the module is only responsible for extending perceived emotion with an evaluation, decision on how to act upon received information must be handled in another component (i.e.Action module)

4.1.3. Action

The objective of theActioncomponent is to help system per- form at its best under existing circumstances, including emotional state of the humans involved in its operation. Emotional state of an individual fluctuates under influence of numerous factors, however actions of emotion aware system are primarily directed to the system itself and do not intend to influence of alter emotional state of the human.

The complexity of action will strongly depend on actual use case, and maturity of the implementation. At an early stage, an action may be as simple as decision to log perceived emotional state. As system matures and acquires more knowledge, action component may instruct system to modify own parameters, e.g.

HMI, in order to adapt to new conditions.

4.1.4. Privacy and security

Privacy and security are compulsory properties for smart manufacturing systems nowadays. Emotional state of a human is rather sensitive information, and often requires use of other sensitive data for its identification. Security policies must de- fine practices to data access and handling within the emotion awareness lattice and across smart manufacturing system components. Privacy implementation must be compliant with legal policies as well as carefully investigate necessity of use of per- sonal data and opt for anonymity where possible. Privacy aspect poses particular challenge because degree to which same emotion manifests in different people varies significantly, and is less subject to cultural background [11], but rather to personality.

Therefore accuracy of emotion inference depends significantly on possibility to train the inference engine to build personalised emotion profiles.

4.2. Architecture for Emotion-aware Digital Twin

As highlighted in [4], there are three components, that con- stitute the technical core of a DT. These are: information model, data processing, and communication. The first two components belong to the DT, while communication component bridges DT and its physical counterpart.

(7)

Fig. 2. Extended view on core functions of emotion-aware digital twin:Perception,Interpretation,Action.

In order for a DT to fulfil its core purpose, physical object must be capable of sensing and communicating sensed data to DT. Perception function in an EADT can be approached in two ways. In first case, physical object would sense emotion symptoms and communicate them to the DT for further processing.

The second approach is to complete perception within the physical object and pass the inferred emotion over to the DT.

Emotion interpretation is the process of assessment of the captured emotion against conditions considered optimal for the physical system performance. This functionality must be fully handled by DT. For this purpose DT may rely on own reference model, as well as employ additional information sources available in the smart manufacturing systems in order to ensure accurate inference.

Finally, an action may be initiated by the DT, once emotion is given an evaluation. The action may be as simple as decision to log or not log captured emotion, and in advanced scenario DT will communicate with physical object and other smart manufacturing components in order to initiate adjustments aiming for improvement in performance under newly learned operational conditions.

A scenario, where EADT helps sustain and enhance human’s performance (i.e. UC3 from 2.2) will be used to illustrate each function. In this example the focus on emotion awareness re- garding a particular individual interacting with smart manufacturing systems.

Smart manufacturing system is complex, and individual in- teracts with it in numerous ways. Those interactions are distributed in time and space. Having the objective to sustain performance and/or foster creativity EADT must be able to perceive emotional state throughout all the interactions as well as maintain a log allowing to detect and address long-term effects.

For this setupperceptionwill be distributed between system components, and a dedicated model will be used to perceive emotion in a particular point in time. For example in shop-floor environment wearable devices assisting job performance, interaction patterns from HMIs of machines would source perception. While tasks performed at the desk would allow to use fa-

cial expression and user input patterns for emotion detection.

For each set of inputs, a separate emotion inference module must be implemented, allowing to keep up with emotional state of an employee at different interaction points.

Emotioninterpretationand followingactionwill be required to operate both in real-time manner and off-line, in terms of user’s interaction with the system. The real-time component will concern with evaluation of current emotion with respect to short-term performance KPIs, log the event and initiate imme- diate action if possible: for example suggest a posture change, adjust lighting at work place, change visual and text interfaces to ensure adequate degree of perception for the job.

EMI performing off-line emotion interpretation will evaluate available emotion log associated with various tasks and related performance KPIs to ensure performance consistency and provide feedback to task allocation, recommending higher or lower diversity in assignments or promoting specific jobs. In case off- line or on-demand operation, Action Module may be engaged automatically, manually or by a client component once interpretation is complete.

In such a scenario additionally to common requirementspri- vacyandsecuritypractices must assist implementation of user data consistency of the distributed solution.

5. Conclusions and future work

Digital manufacturing employs different enabling technologies to represent various aspects of product and its manufacturing processes to be able proactively make better, i.e. informative decisions. Digital twins is the recently developed approach for having a representation of manufacturing site components in digital space allowing to give feedback and predict development of the product or the system. High-fidelity of representation and ability to communicate with its physical counterpart are critical properties of a digital twin.

While manufacturing industry transitions to being highly digitalised on every level, human remains an important contrib-

(8)

utor to its operation. Therefore, adequate representation of human contributes to completeness and accuracy of digital twins for manufacturing. In order to address the skill requirements for employees of future factories, such representation must also cover emotional state.

Being important tool for digital manufacturing, digital twins naturally employ technologies and tools considered to be main enablers for future manufacturing systems. As consequence, same technologies become effortlessly available for implementation of emotion awareness capability, making digital twins most feasible component to host this functionality.

A high level architecture was proposed describing main functions an EADT must implement. System must realize three behaviours in order to allow emotion awareness: perception, interpretation, andaction. These behaviours serve to capture and understand an emotion, and adjust system’s performance if needed. Execution of each behaviour must comply with privacy and security policies set within the system.

Such architecture allows implementing systems enabling human to keep the locus of control by better comprehending, communicating relevant data and promoting situational awareness.

Majority of challenges related to implementation of proposed approach concern with practical aspects of realisation of privacy and security requirements, finding suitable formats and tools for emotion data handling, as well as availability of required input data and bandwidth for handling it in the desired manner.

Due to variations in requirements in each of the use cases described in this paper, proposed design will undergo re- evaluation prior to implementation phase. While high-level architecture ensures clear phases and scopes of operation for an EADT, design of each individual component implementing high-level function will require revision upon implementation with objective to meet requirements of scenario in question without blocking other scenarios.

References

[1] V. Alc´acer, V. Cruz-Machado, Scanning the Industry 4.0: A Literature Re- view on Technologies for Manufacturing Systems, Engineering Science and Technology, an International Journal 22 (3) (2019) 899–919.

[2] A. Lobov, Smart manufacturing systems: Climbing the DIKW pyramid, in:

Proceedings of 44th Annual Conference of the IEEE Industrial Electronics Society - IECON 2018, 2018, pp. 4730–4735.

[3] E. H. D. R. Da Silva, J. Angelis, E. P. Lima, In pursuit of digital manufacturing, Procedia Manufacturing 28 (2019) 63–69.

[4] Y. Lu, C. Liu, K. I. Wang, H. Huang, X. Xu, Digital Twin-driven smart manufacturing: Connotation, reference model, applications and research issues, Robotics and Computer-Integrated Manufacturing 61 (August 2019) (2020) 101837.

[5] M. Grieves, J. Vickers, Transdisciplinary Perspectives on Complex Sys- tems: New Findings and Approaches, Springer International Publishing, 2017, Ch. Digital Twin: Mitigating Unpredictable, Undesirable Emergent Behavior in Complex Systems, pp. 85–113.

[6] R. W. Picard, Affective computing, Tech. Rep. 321, M.I.T Media Labora- tory Perceptual Computing Section (1995).

[7] S. E. Zohora, A. M. Khan, A. K. Srivastava, N. G. Nguyen, N. Dey, A Study of the State of the Art in Synthetic Emotional Intelligence in Affec- tive Computing, International Journal of Synthetic Emotions 7 (1) (2016) 1–12.

[8] S. Jain, K. Asawa, Modeling of emotion elicitation conditions for a cognitive-emotive architecture, Cognitive Systems Research 55 (2019) 60–

76.

[9] J. Hu, J. Li, Z. Wei, Y. Liu, A framework for modeling emotion in synthetic agent, 2014 IEEE Chinese Guidance, Navigation and Control Conference, CGNCC 2014 (2015) 1151–1154.

[10] J. L. Salmeron, Simulating synthetic emotions with fuzzy grey cognitive maps, Advances in Intelligent Systems and Computing 316 (December) (2015) 39–46.

[11] J. Vallverd´u, D. Casacuberta, T. Nishida, Y. Ohmoto, S. Moran, S. L´azare, From Computational Emotional Models to HRI, International Journal of Robotics Applications and Technologies 1 (2) (2014) 11–25.

[12] M. Schmitz, B. C. Fehringer, M. Akbal, Expressing emotions with synthetic affect bursts, in: Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play, CHI PLAY ’15, 2015, pp. 91–95.

[13] E. Paule, Universal Facial Expressions of Emotions, California Mental Health Research Digest 8 (4) (1970) 151–158.

[14] J. Tang, Y. Zhang, J. Sun, J. Rao, W. Yu, Y. Chen, A. C. M. Fong, Quanti- tative study of individual emotional states in social networks, IEEE Trans- actions on Affective Computing 3 (2) (2012) 132–144.

[15] B. Rienties, B. A. Rivers, Measuring and understanding learner emotions:

Evidence and prospects, Tech. rep., Learning Analytics Community Ex- change (2014).

[16] C. Martinho, M. Gomes, A. Paiva, Intelligent Virtual Agents, Springer Berlin Heidelberg, 2003, Ch. Synthetic Emotension, pp. 57–61.

[17] M. W.Grieves, Complex Systems Engineering: Theory and Practice, Amer- ican Institute of Aeronautics and Astronautics, 2019, Ch. Virtually Intelli- gent Product Systems: Digital and Physical Twins, pp. 175–200.

[18] Q. Qi, F. Tao, Digital Twin and Big Data Towards Smart Manufacturing and Industry 4.0: 360 Degree Comparison, IEEE Access 6 (2018) 3585–3593.

[19] J. Guo, N. Zhao, L. Sun, S. Zhang, Modular based flexible digital twin for factory design, Journal of Ambient Intelligence and Humanized Comput- ing 10 (3) (2019) 1189–1200.

[20] Q. Qi, F. Tao, A Smart Manufacturing Service System Based on Edge Computing, Fog Computing, and Cloud Computing, IEEE Access 7 (2019) 86769–86777.

[21] N. Ahmad, N. A. A. Seman, A. Shamsuddin, Industry 4.0 Implications on Human Capital : A Review, Journal for Studies in Management and Planning 4 (Special Issue-13) (2019) 221–235.

[22] P. M. Niedenthal, J. B. Halberstadt, M. B. Setterlund, Being Happy and Seeing ”Happy”: Emotional State Mediates Visual Word Recognition, Cognition and Emotion 11 (4) (1997) 403–432.

[23] T. Brosch, K. R. Schrerer, D. Grandjean, D. Sander, The impact of emotion on perception, attention, memory, and decision-making, Swiss Medi- cal Weekly (2013).

[24] X. Zhang, B. Zuo, K. Erskine, T. Hu, Feeling light or dark? Emotions affect perception of brightness, Journal of Environmental Psychology 47 (2016) 107–111.

[25] M. Nykänen, V. Puro, M. Tiikkaja, H. Kannisto, E. Lantto, F. Simpura, J. Uusitalo, K. Lukander, T. Räsänen, A.-M. Teperi, Evaluation of the ef- ficacy of a virtual reality-based safety training and human factors training method: study protocol for a randomised-controlled trial, Injury Prevention (2019) injuryprev–2019–043304.

[26] S. Draˇce, F. Ric, The effect of emotions on risk perception: Experimental evaluation of the affective tendencies framework, Psihologija 45 (4) (2012) 409–416.

[27] N. Hao, H. Xue, H. Yuan, Q. Wang, M. A. Runco, Enhancing creativity:

Proper body posture meets proper emotion, Acta Psychologica 173 (2017) 32–40.

[28] S. Droit-Volet, W. H. Meck, How emotions colour our perception of time, Trends in Cognitive Sciences 11 (12) (2007) 504–513.

[29] I. Graessler, A. Poehler, Intelligent control of an assembly station by inte- gration of a digital twin for employees into the decentralized control system, Procedia Manufacturing 24 (2018) 185–189.

[30] I. Yang, A. Kelly, The positive outcomes of ’Socially Sharing Negative Emotions’ in workteams: A conceptual exploration, European Manage- ment Journal 34 (2) (2016) 172–181.