本研究では、過去の学習経験に基づいて、実世界に適した動作をリアルタイムに予測することで、未学習の環境や作業対象物に対応することが可能な深層予測学習を用いた双腕ロボットによる柔軟物体操作を提案する。柔軟物体操作としてタオル掛けタスクを対象に、ロボットは机の上に置かれたタオルを把持し、物干し竿にかける動作を実現する。ロボットは視覚運動情報に基づき近未来の状況を予測し、現実との誤差を最小にするように動作を生成し、学習時と現実の差を許容しながらリアルタイムに動作を調整し続けることで、未学習の状況下でも柔軟に作業することが可能である。

Abstract

The brain attenuates its responses to self-produced exteroceptions (e.g., we cannot tickle ourselves). Is this phenomenon, known as sensory attenuation, enabled innately, or acquired through learning? Here, our simulation study using a multimodal hierarchical recurrent neural network model, based on variational free-energy minimization, shows that a mechanism for sensory attenuation can develop through learning of two distinct types of sensorimotor experience, involving self-produced or externally produced exteroceptions. For each sensorimotor context, a particular free-energy state emerged through interaction between top-down prediction with precision and bottom-up sensory prediction error from each sensory area. The executive area in the network served as an information hub. Consequently, shifts between the two sensorimotor contexts triggered transitions from one free-energy state to another in the network via executive control, which caused shifts between attenuating and amplifying prediction-error-induced responses in the sensory areas. This study situates emergence of sensory attenuation (or self-other distinction) in development of distinct free-energy states in the dynamic hierarchical neural system.

Robots need robust models to effectively perform tasks that humans do on a daily basis. These models often require substantial developmental costs to maintain because they need to be adjusted and adapted over time. Deep reinforcement learning is a powerful approach for acquiring complex real-world models because there is no need for a human to design the model manually. Furthermore, a robot can establish new motions and optimal trajectories that may not have been considered by a human. However, the cost of learning is an issue because it requires a huge amount of trial and error in the real world. Here, we report a method for realizing complicated tasks in the real world with low design and teaching costs based on the principle of prediction error minimization. We devised a module integration method by introducing a mechanism that switches modules based on the prediction error of multiple modules. The robot generates appropriate motions according to the door’s position, color, and pattern with a low teaching cost. We also show that by calculating the prediction error of each module in real time, it is possible to execute a sequence of tasks (opening door outward and passing through) by linking multiple modules and responding to sudden changes in the situation and operating procedures. The experimental results show that the method is effective at enabling a robot to operate autonomously in the real world in response to changes in the environment.

Deep robotic learning by learning from demonstration allows robots to mimic a given demonstration and generalize their performance to unknown task setups. However, this generalization ability is heavily affected by the number of demonstrations, which can be costly to manually generate. Without sufficient demonstrations, robots tend to overfit to the available demonstrations and lose the robustness offered by deep learning. Applying the concept of motor babbling – a process similar to that by which human infants move their bodies randomly to obtain proprioception – is also effective for allowing robots to enhance their generalization ability. Furthermore, the generation of babbling data is simpler than task-oriented demonstrations. Previous researches use motor babbling in the concept of pre-training and fine-tuning but have the problem of the babbling data being overwritten by the task data. In this work, we propose an RNN-based robot-control framework capable of leveraging targetless babbling data to aid the robot in acquiring proprioception and increasing the generalization ability of the learned task data by learning both babbling and task data simultaneously. Through simultaneous learning, our framework can use the dynamics obtained from babbling data to learn the target task efficiently. In the experiment, we prepare demonstrations of a block-picking task and aimless-babbling data. With our framework, the robot can learn tasks faster and show greater generalization ability when blocks are at unknown positions or move during execution.

We propose a tool-use model that enables a robot to act toward a provided goal. It is important to consider features of the four factors; tools, objects actions, and effects at the same time because they are related to each other and one factor can influence the others. The tool-use model is constructed with deep neural networks (DNNs) using multimodal sensorimotor data; image, force, and joint angle information. To allow the robot to learn tool-use, we collect training data by controlling the robot to perform various object operations using several tools with multiple actions that leads different effects. Then the tool-use model is thereby trained and learns sensorimotor coordination and acquires relationships among tools, objects, actions and effects in its latent space. We can give the robot a task goal by providing an image showing the target placement and orientation of the object. Using the goal image with the tool-use model, the robot detects the features of tools and objects, and determines how to act to reproduce the target effects automatically. Then the robot generates actions adjusting to the real time situations even though the tools and objects are unknown and more complicated than trained ones.

Neurodevelopmental disorders are characterized by heterogeneous and non-specific nature of their clinical symptoms. In particular, hyper- and hypo-reactivity to sensory stimuli are diagnostic features of autism spectrum disorder and are reported across many neurodevelopmental disorders. However, computational mechanisms underlying the unusual paradoxical behaviors remain unclear. In this study, using a robot controlled by a hierarchical recurrent neural network model with predictive processing and learning mechanism, we simulated how functional disconnection altered the learning process and subsequent behavioral reactivity to environmental change. The results show that, through the learning process, long-range functional disconnection between distinct network levels could simultaneously lower the precision of sensory information and higher-level prediction. The alteration caused a robot to exhibit sensory-dominated and sensory-ignoring behaviors ascribed to sensory hyper- and hypo-reactivity, respectively. As long-range functional disconnection became more severe, a frequency shift from hyporeactivity to hyperreactivity was observed, paralleling an early sign of autism spectrum disorder. Furthermore, local functional disconnection at the level of sensory processing similarly induced hyporeactivity due to low sensory precision. These findings suggest a computational explanation for paradoxical sensory behaviors in neurodevelopmental disorders, such as coexisting hyper- and hypo-reactivity to sensory stimulus. A neurorobotics approach may be useful for bridging various levels of understanding in neurodevelopmental disorders and providing insights into mechanisms underlying complex clinical symptoms. (c) 2021 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

In many robotics studies, deep neural networks (DNNs) are being actively studied due to their good performance. However, existing robotic techniques and DNNs have not been systematically integrated, and packages for beginners are yet to be developed. In this study, we proposed a basic educational kit for robotic system development with DNNs. Our goal was to educate beginners in both robotics and machine learning, especially the use of DNNs. Initially, we required the kit to (1) be easy to understand, (2) employ experience-based learning, and (3) be applicable in many areas. To clarify the learning objectives and important parts of the basic educational kit, we analyzed the research and development (R&D) of DNNs and divided the process into three steps of data collection (DC), machine learning (ML), and task execution (TE). These steps were configured under a hierarchical system flow with the ability to be executed individually at the development stage. To evaluate the practicality of the proposed system flow, we implemented it for a physical robotic grasping system using robotics middleware. We also demonstrated that the proposed system can be effectively applied to other hardware, sensor inputs, and robot tasks.

Robotic applications require both correct task performance and compensation for undefined behaviors. Although deep learning is a promising approach to perform complex tasks, the response to undefined behaviors that are not reflected in the training dataset remains challenging. In a human-robot collaborative task, the robot may adopt an unexpected posture due to collisions and other unexpected events. Therefore, robots should be able to recover from disturbances for completing the execution of the intended task. We propose a compensation method for undefined behaviors by switching between two controllers. Specifically, the proposed method switches between learning-based and model-based controllers depending on the internal representation of a recurrent neural network that learns task dynamics. We applied the proposed method to a pick-And-place task and evaluated the compensation for undefined behaviors. Experimental results from simulations and on a real robot demonstrate the effectiveness and high performance of the proposed method.

Selection of appropriate tools and use of them when performing daily tasks is a critical function for introducing robots for domestic applications. In previous studies, however, adaptability to target objects was limited, making it difficult to accordingly change tools and adjust actions. To manipulate various objects with tools, robots must both understand tool functions and recognize object characteristics to discern a tool-object-action relation. We focus on active perception using multimodal sensorimotor data while a robot interacts with objects, and allow the robot to recognize their extrinsic and intrinsic characteristics. We construct a deep neural networks (DNN) model that learns to recognize object characteristics, acquires tool-object-action relations, and generates motions for tool selection and handling. As an example tool-use situation, the robot performs an ingredients transfer task, using a turner or ladle to transfer an ingredient from a pot to a bowl. The results confirm that the robot recognizes object characteristics and servings even when the target ingredients are unknown. We also examine the contributions of images, force, and tactile data and show that learning a variety of multimodal information results in rich perception for tool use.

In environments where various tasks are sequentially given to deep neural networks (DNNs), training methods are needed that enable DNNs to learn the given tasks continuously. A DNN is typically trained by a single dataset, and continuous learning of subsequent datasets causes the problem of catastrophic forgetting. Previous studies have reported results for consolidation learning methods in recognition tasks and reinforcement learning problems. However, those methods were validated on only a few examples of predictive learning for time series. In this study, we applied elastic weight consolidation (EWC) and pseudo-rehearsal to the predictive learning of time series and compared their learning results. Evaluating the latent space after the consolidation learning revealed that the EWC method acquires properties of the pre-training and subsequent datasets with the same distribution, and the pseudo-rehearsal method distinguishes the properties and acquires them with different distributions.

This study proposed a hypothesis regarding the emergence of object-oriented action via tactile-based curiosity. The hypothesis is such that a curious exploration driven by tactile sensation leads tactile-rich object-oriented actions, while there are no explicit rewards or other designated intentional purposes. Experiments were with the curiosity model named the disagreement model from the reinforcement learning research field and with a simple physics robotic simulation with visual and tactile sensory information. The experimental results indicated that the tactile sensation induces object-oriented actions such as hitting and pecking by the body parts that have tactile sensors. We deduced that the hypothesis could be extended to discussions regarding the acquisition of dexterous skillful object manipulation in human development.

We propose a deep learning model for a robot to wipe 3D-objects. Wiping of 3D-objects requires recognizing the shapes of objects and planning the motor angle adjustments for tracing the objects. Unlike previous research, our learning model does not require pre-designed computational models of target objects. The robot is able to wipe the objects to be placed by using image, force, and arm joint information. We evaluate the generalization ability of the model by confirming that the robot handles untrained cube and bowl shaped-objects. We also find that it is necessary to use both image and force information to recognize the shape of and wipe 3D objects consistently by comparing changes in the input sensor data to the model. To our knowledge, this is the first work enabling a robot to use learning sensorimotor information alone to trace various unknown 3D-shape.

Performing various in-hand manipulation tasks, without learning each individual task, would enable robots to act more versatile, while reducing the effort for training. However, in general it is difficult to achieve stable in-hand manipulation, because the contact state between the fingertips becomes difficult to model, especially for a robot hand with anthropomorphically shaped fingertips. Rich tactile feedback can aid the robust task execution, but on the other hand it is challenging to process high-dimensional tactile information. In the current paper we use two fingers of the Allegro hand, and each fingertip is anthropomorphically shaped and equipped not only with 6-axis force-torque (F/T) sensors, but also with uSkin tactile sensors, which provide 24 tri-axial measurements per fingertip. A convolutional neural network is used to process the high dimensional uSkin information, and a long short-term memory (LSTM) handles the time-series information. The network is trained to generate two different motions ("twist"and "push"). The desired motion is provided as a task-parameter to the network, with twist defined as -1 and push as +1. When values between -1 and +1 are used as the task parameter, the network is able to generate untrained motions in-between the two trained motions. Thereby, we can achieve multiple untrained manipulations, and can achieve robustness with high-dimensional tactile feedback.

The use of tactile information is one of the most important factors for achieving stable in-grasp manipulation. Especially with low-cost robotic hands that provide low-precision control, robust in-grasp manipulation is challenging. Abundant tactile information could provide the required feed-back to achieve reliable in-grasp manipulation also in such cases. In this research, soft distributed 3-axis skin sensors ("uSkin") and 6-axis F/T (force/torque) sensors were mounted on each fingertip of an Allegro Hand to provide rich tactile information. These sensors yielded 78 measurements for each fingertip (72 measurements from the uSkin and 6 measurements from the 6-axis F/T sensor). However, such high-dimensional tactile information can be difficult to process because of the complex contact states between the grasped object and the fingertips. Therefore, a convolutional neural network (CNN) was employed to process the tactile information. In this paper, we explored the importance of the different sensors for achieving in-grasp manipulation. Successful in-grasp manipulation with untrained daily objects was achieved when both 3-axis uSkin and 6-axis F/T information was provided and when the information was processed using a CNN.

Neurodevelopmental disorders, including autism spectrum disorder, have been intensively investigated at the neural, cognitive, and behavioral levels, but the accumulated knowledge remains fragmented. In particular, developmental learning aspects of symptoms and interactions with the physical environment remain largely unexplored in computational modeling studies, although a leading computational theory has posited associations between psychiatric symptoms and an unusual estimation of information uncertainty (precision), which is an essential aspect of the real world and is estimated through learning processes. Here, we propose a mechanistic explanation that unifies the disparate observations via a hierarchical predictive coding and developmental learning framework, which is demonstrated in experiments using a neural network-controlled robot. The results show that, through the developmental learning process, homogeneous intrinsic neuronal excitability at the neural level induced via self-organization changes at the information processing level, such as hyper sensory precision and overfitting to sensory noise. These changes led to multifaceted alterations at the behavioral level, such as inflexibility, reduced generalization, and motor clumsiness. In addition, these behavioral alterations were accompanied by fluctuating neural activity and excessive development of synaptic connections. These findings might bridge various levels of understandings in autism spectrum and other neurodevelopmental disorders and provide insights into the disease processes underlying observed behaviors and brain activities in individual patients. This study shows the potential of neurorobotics frameworks for modeling how psychiatric disorders arise from dynamic interactions among the brain, body, and uncertain environments.

＜文献概要＞脳の計算理論の発展に伴い,それを精神疾患の病態理解に応用する試みへの期待が,近年,高まっている。特に,脳の予測符号化モデルに基づいた研究では,感覚情報の不確実性を推定する神経機能の変調と精神疾患との関連を示唆する概念的な仮説が提示されてきたが,実際の臨床で観察される行動レベルでの病像との間にはギャップが存在している。本論文では,神経ロボティクスの手法を用いることで,実験的にこの理論と臨床像との橋渡しを行った研究を紹介する。実験では,自閉スペクトラム症に特徴的な限定された反復的な行動様式や行為の停止といった多様な症状が,不確実性の過小評価と過大評価による異なるプロセスから生じ得ることが示され,精神症状のmultifinal性とequifinal性の一側面が捉えられた。また,病態理解にとどまらず,症状改善のための環境調整的な介入への理論的示唆といった,計算論的精神医学の臨床研究への貢献可能性が示された。

For an autonomous robot to flexibly move in response to various tasks or environmental changes, an attention mechanism is required that is based on the robot's behavioral experience. In this paper, we visualize how attention is acquired inside a neural network learned using supervised learning and describe how to acquire a suitable representation for performing a task. Our experimental evaluation shows that the attention was automatically acquired for objects that are needed to perform tasks by learning the time-series of both vision and motor information rather than only vision information. By multimodal learning, the attention is robust against unlearned conditions which background changes or obstacles.

We propose a sensorimotor dynamical system model for pouring unknown liquids. With our system, a robot holds and shakes a bottle to estimate the characteristics of the contained liquid, such as viscosity and fill level, without calculating to determine their parameters. Next, the robot pours a specified amount of the liquid into another container. The system needs to integrate information on the robot's actions, the liquids, the container, and the surrounding environment to perform the estimation and execute a continuous pouring motion using the same model. We use deep neural networks (DNN) to construct the system. The DNN model repeats prediction and execution of the actions to be taken in the next time step based on the input sensorimotor data, including camera images, force sensor data, and joint angles. At the same time, the DNN model acquires liquid characteristics in the internal state. We confirmed that the DNN model can control the robot to pour a desired amount of liquid with unknown viscosity and fill level.

In this paper we present a neural architecture to learn a bi-directional mapping between actions and language. We implement a Multiple Timescale Long Short-Term Memory (MT-LSTM) network comprised of 7 layers with different timescale factors, to connect actions to language without explicitly learning an intermediate representation. Instead, the model self-organizes such representations at the level of a slow-varying latent layer, linking action branch and language branch at the center. We train the model in a bi-directional way, learning how to produce a sentence from a certain action sequence input and, simultaneously, how to generate an action sequence given a sentence as input. Furthermore we show this model preserves some of the generalization behaviour of Multiple Timescale Recurrent Neural Networks (MTRNN) in generating sentences and actions that were not explicitly trained. We compare this model with a number of different baseline models, confirming the importance of both the bi-directional training and the multiple timescales architecture. Finally, the network was evaluated on motor actions performed by an iCub robot and their corresponding letter-based description. The results of these experiments are presented at the end of the paper.

As the complexity of the robot's tasks increases, we can consider many general tasks in a compound form that consists of shorter tasks. Therefore, for robots to generate various tasks, they need to be able to execute shorter tasks in succession, appropriately to the situation. With the design principle to construct the architecture for robots to execute complex tasks compounded with multiple subtasks, this study proposes a visuomotor-control framework with the characteristics of a state machine to train shorter tasks as sensorimotor units. The design procedure of training framework consists of 4 steps: (1) segment entire task into appropriate subtasks, (2) define subtasks as states and transitions in a state machine, (3) collect subtasks data, and (4) train neural networks: (a) autoencoder to extract visual features, (b) a single recurrent neural network to generate subtasks to realize a pseud-state-machine model with a constraint in hidden values. We implemented this framework on two different robots to allow their performance of repetitive tasks with error-recovery motion, subsequently, confirming the ability of the robot to switch the sensorimotor units from visual input at the attractors of the hidden values created by the constraint.

Drawing is a representative human cognitive ability and may mirror cognitive characteristics including those associated with psychiatric symptoms. Therefore, analysis of drawing data collected from various populations such as healthy people and psychiatric patients may be beneficial for better understanding human cognition. However, collecting such large-scale data about the relationship between drawing and cognitive/personality traits offline-in a laboratory-is a difficult issue. To overcome this issue, we devised a novel experimental paradigm involving a goal-directed drawing task conducted online-on the eb-with participants recruited via a crowdsourcing platform. With the assistance of 1155 participants with differing levels of psychiatric symptoms, we collected a total of 194, 040 trajectory data and answers to seven different self-report psychiatric symptom questionnaires comprising 181 items. We visualized the collected trajectory data and performed an exploratory factor analysis on the correlation matrix of the psychiatric symptom questionnaire items. Our results suggest that there were associations between psychiatric symptoms represented by specific psychiatric factors and atypical behavior observed while performing the goal-directed drawing task. This indicates the efficacy of a dimensional approach to large-scale online experiments with respect to clinical psychiatry.

Casual conversations involving multiple speakers and noises from surrounding devices are common in everyday environments, which degrades the performances of automatic speech recognition systems. These challenging characteristics of environments are the target of the CHiME-5 challenge. By employing a convolutional neural network (CNN)-based multichannel end-to-end speech recognition system, this study attempts to overcome the presents difficulties in everyday environments. The system comprises of an attention-based encoder-decoder neural network that directly generates a text as an output from a sound input. The multichannel CNN encoder, which uses residual connections and batch renormalization, is trained with augmented data, including white noise injection. The experimental results show that the word error rate is reduced by 8.5% and 0.6% absolute from a single channel end-to-end and the best baseline (LF-MMI TDNN) on the CHiME-5 corpus, respectively.

Studies suggest that the difference of the sensorimotor events can be recorded with the fast- and slower-changing neural activities in the hierarchical brain areas, in which they have bi-directional connections. The slow-changing representations attempt to predict the activities on the faster level by relaying categorized sensorimotor events. On the other hand, the incoming sensory information corrects such event-based prediction on the higher level by the novel or surprising signal. From this motivation, we propose a predictive hierarchical artificial neural network model which is implemented the differentiated temporal parameters for neural updates. Also, both the fast- and slow-changing neural activities are modulated by the active motor activities. This model is examined in the driving dataset, recorded in various events, which incorporates the image sequences and the ego-motion of the vehicle. Experiments show that the model encodes the driving scenarios on the higher-level where the neuron recorded the long-term context.

We propose a tool-use model that can select tools that require neither labeling nor modeling of the environment and actions. With this model, a robot can choose a tool by itself and perform the operation that matches a human command and the environmental situation. To realize this, we use deep learning to train sensory motor data recorded during tool selection and tool use as experienced by a robot. The experience includes two types of selection, namely according to function and according to size, thereby allowing the robot to handle both situations. For evaluation, the robot is required to generate motion either in an untrained situation or using an untrained tool. We confirm that the robot can choose and use a tool that is suitable for achieving the target task.

We propose a tool-use model that can detect the features of tools, target objects, and actions from the provided effects of object manipulation. We construct a model that enables robots to manipulate objects with tools, using infant learning as a concept. To realize this, we train sensory-motor data recorded during a tool-use task performed by a robot with deep learning. Experiments include four factors: (1) tools, (2) objects, (3) actions, and (4) effects, which the model considers simultaneously. For evaluation, the robot generates predicted images and motions given information of the effects of using unknown tools and objects. We confirm that the robot is capable of detecting features of tools, objects, and actions by learning the effects and executing the task.

Understanding and grounding human commands with natural languages have been a fundamental requirement for service robotic applications. Although there have been several attempts toward this goal, the bottleneck still exists to store and process the corpora of natural language in an interaction system. Currently, the neural- and statistical-based (N&S) natural language processing have shown potential to solve this problem. With the availability of large data-sets nowadays, these processing methods are able to extract semantic relationships while parsing a corpus of natural language (NL) text without much human design, compared with the rule-based language processing methods. In this paper, we show that how two N&S based word embedding methods, called Word2vec and GloVe, can be used in natural language understanding as pre-training tools in a multi-modal environment. Together with two different multiple time-scale recurrent neural models, they form hybrid neural language understanding models for a robot manipulation experiment.

In this paper, we discuss about the learning performance of dynamics learning tree (DLT) while mainly focusing on the implementation on robot arms. We propose an input-order-designing method for DLT. DLT has been applied to the modeling of boat, vehicle, and humanoid robot. However, the relationship between the input order and the performance of DLT has not been investigated. In the proposed method, a developer is able to design an effective input order intuitively. The proposed method was validated in the model learning tasks on a simulated robot manipulator, a real robot manipulator, and a simulated vehicle. The first/second manipulator was equipped with flexible arm/finger joints that made uncertainty around the trajectories of manipulated objects. In all of the cases, the proposed method improved the performance of DLT.

Studies suggest that, within the hierarchical architecture, the topological higher level possibly represents the scenarios of the current sensory events with slower changing activities. They attempt to predict the neural activities on the lower level by relaying the predicted information after the scenario of the sensorimotor event has been determined. On the other hand, the incoming sensory information corrects such prediction of the events on the higher level by the fast-changing novel or surprising signal. From this point, we propose a predictive hierarchical artificial neural network model that examines this hypothesis on neurorobotic platforms. It integrates the perception and action in the predictive coding framework. Moreover, in this neural network model, there are different temporal scales of predictions existing on different levels of the hierarchical predictive coding architecture, which defines the temporal memories in recording the events occurring. Also, both the fast- and the slow-changing neural activities are modulated by the motor action. Therefore, the slow-changing neurons can be regarded as the representation of the recent scenario which the sensorimotor system has encountered. The neurorobotic experiments based on the architecture were also conducted.

Recently, the importance of the application of computational models utilized in cognitive neuroscience to psychiatric disorders has been recognized. This study utilizes a recurrent neural network model to test aberrant sensory precision, a normative theory of autism spectrum disorder. We particularly focus on the effects of increased and decreased sensory precision on adaptive behavior based on a prediction error minimization mechanism. To distinguish dysfunction at the behavioral and network levels, we employ a humanoid robot driven by a neural network and observe ball-playing interactions with a human experimenter. Experimental results show that behavioral rigidity characteristic of autism spectrum disorder-including stopping movement and repetitive behavior-was generated from both increased and decreased sensory precision, but through different processes at the network level. These results may provide a system-level explanation of different types of behavioral rigidity in psychiatric diseases such as compulsions and stereotypies. The results also support a system-level model for autism spectrum disorder that suggests core deficits in estimating the uncertainty of sensory evidence.

We propose a tool-body assimilation model that considers grasping during motor babbling for using tools. A robot with tool-use skills can be useful in human–robot symbiosis because this allows the robot to expand its task performing abilities. Past studies that included tool-body assimilation approaches were mainly focused on obtaining the functions of the tools, and demonstrated the robot starting its motions with a tool pre-attached to the robot. This implies that the robot would not be able to decide whether and where to grasp the tool. In real life environments, robots would need to consider the possibilities of tool-grasping positions, and then grasp the tool. To address these issues, the robot performs motor babbling by grasping and nongrasping the tools to learn the robot's body model and tool functions. In addition, the robot grasps various parts of the tools to learn different tool functions from different grasping positions. The motion experiences are learned using deep learning. In model evaluation, the robot manipulates an object task without tools, and with several tools of different shapes. The robot generates motions after being shown the initial state and a target image, by deciding whether and where to grasp the tool. Therefore, the robot is capable of generating the correct motion and grasping decision when the initial state and a target image are provided to the robot.

We suggest that different behavior generation schemes, such as sensory reflex behavior and intentional proactive behavior, can be developed by a newly proposed dynamic neural network model, named stochastic multiple timescale recurrent neural network (S-MTRNN). The model learns to predict subsequent sensory inputs, generating both their means and their uncertainty levels in terms of variance (or inverse precision) by utilizing its multiple timescale property. This model was employed in robotics learning experiments in which one robot controlled by the S-MTRNN was required to interact with another robot under the condition of uncertainty about the other's behavior. The experimental results show that self-organized and sensory reflex behavior-based on probabilistic prediction-emerges when learning proceeds without a precise specification of initial conditions. In contrast, intentional proactive behavior with deterministic predictions emerges when precise initial conditions are available. The results also showed that, in situations where unanticipated behavior of the other robot was perceived, the behavioral context was revised adequately by adaptation of the internal neural dynamics to respond to sensory inputs during sensory reflex behavior generation. On the other hand, during intentional proactive behavior generation, an error regression scheme by which the internal neural activity was modified in the direction of minimizing prediction errors was needed for adequately revising the behavioral context. These results indicate that two different ways of treating uncertainty about perceptual events in learning, namely, probabilistic modeling and deterministic modeling, contribute to the development of different dynamic neuronal structures governing the two types of behavior generation schemes.

This study demonstrates that the prediction error minimization (PEM) mechanism can account for the emergence of reciprocal interaction between two cognitive agents. During interactive processes, alternation of forming and deforming interactions may be triggered by various internal and external causes. We focus in particular on external causes derived from a dynamic and uncertain environment. Two small humanoid robots controlled by an identical dynamic neural network model using the PEM mechanism were trained to achieve a set of coherent ball-playing interactions between them. The two robots predict each other in a top-down way while they try to minimize the prediction errors derived from the unstable ball dynamics or the external cause in a bottom-up way by using the PEM mechanism. The experimental results showed that switching among the set of trained interactive ball plays between the two robots appears spontaneously. The analysis clarified how each complementary behavior can be generated via mutual adaptation between the two robots by undertaking top-down and bottom-up interaction in each individual dynamic neural network model by using the PEM mechanism.

Human communicative behavior is both dynamic and bidirectional. This study aims to analyze such behavior by conducting imitative interactions between human subjects and a humanoid robot that has a neuro-dynamical system. For this purpose, we take a robot-centered approach in which the change in robot performance according to difference in human partner is analyzed, rather than adopting the typical human-centered approach. A small humanoid robot equipped with a neuro-dynamical system learns imitative arm movement patterns and interacts with humans after the learning process. We analyze the interactive phenomena by different methods, including principal component analysis and use of a recurrence plot. Through this analysis, we demonstrate that different classes of interactions can be observed in the contextual dynamics of the neuro-dynamical system.

This study proposes the use of a deep neural network to localize a sound source using an array of microphones in a reverberant environment. During the last few years, applications based on deep neural networks have performed various tasks such as image classification or speech recognition to levels that exceed even human capabilities. In our study, we employ deep residual networks, which have recently shown remarkable performance in image classification tasks even when the training period is shorter than that of other models. Deep residual networks are used to process audio input similar to multiple signal classification (MUSIC) methods. We show that with end-to-end training and generic preprocessing, the performance of deep residual networks not only surpasses the block level accuracy of linear models on nearly clean environments but also shows robustness to challenging conditions by exploiting the time delay on power information.

<p>In this study, we propose a framework for object recognition and reaching motion in robot systems with robot middleware. There exist many elemental technologies for robot manipulation system such as object recognition and reaching. However, each technology is not systematically integrated for component-based development as open and common frameworks are undeveloped. In order to realize systematic integration of object recognition and reaching motion, we built a framework of robot middleware and proposed it as open framework. This framework is oriented reusable, exchangeable and extensible system and does not depend on specific robot middleware platforms. We implemented the proposed framework in a pick-and-place system with robot arm, and validate the system working.</p>

<p>In recent years, several robot simulators and open source game engines have been developed. Open source game engines such as Unity and Unreal Engine 4 help to create the simulation environment. Therefore, we implemented a simulator environment by connecting a game engine and a robot middleware. The requirements for the simulator are follows: (1) To create a simulation environment with the high degree of freedom. (2) To support the multiple OS. In this study, we selected Unity and OpenRTM-aist. To assess the machine learning performance, the training data can be collected and the learned model can be verified in the simulator environment. As an example of the implementation of the simulation environment using this framework, we developed a simulator for autonomous driving systems.</p>

<p>This paper presents an autonomous mobile system using E2E learning. We examined the autonomous mobile system which could move in real environment for service robot and autonomous mobile vehicles. As a result, we proposed a system using MTRNN for E2E learning method of the autonomous mobile system. In addition, we developed the mobile robot system for the experiments. We suggested the machine learning simulator which could collect enormous learning data at low cost and developed it. By the above, we developed learning system of the autonomous mobile system by E2E learning.</p>

<p>Robots working in the real environment need to respond to necessary sensory inputs. However, if the sensory inputs are not necessary for action generation, robots need to stably generate action without being affected by unnecessary sensory inputs. To realize such adaptive action generation against situational changes, robots should automatically decide how much sensory inputs are necessary for action generation. In this research, we propose a method which automatically decides the ratio of actual and predicted sensory inputs based on predicted sensory uncertainty. As a result of robot experiments, the robot with proposed mechanism could conduct adaptive action generation against situational changes.</p>

A model was developed to allow a mobile robot to label the areas of a typical domestic room, using raw sequential visual and motor data, no explicit information on location was provided, and no maps were constructed. The model comprised a deep autoencoder and a recurrent neural network. The model was demonstrated to (1) learn to correctly label areas of different shapes and sizes, (2) be capable of adapting to changes in room shape and rearrangement of items in the room, and (3) attribute different labels to the same area, when approached from different angles. Analysis of the internal representations of the model showed that a topological structure corresponding to the room structure was self-organized as the trajectory of the internal activations of the network.

In this paper, we propose a method to dynamically modulate the input state of recurrent neural networks (RNNs) so as to realize flexible and robust robot behavior. We employ the so-called stochastic continuous-time RNN (S-CTRNN), which can learn to predict the mean and variance (or uncertainty) of subsequent sensorimotor information. Our proposed method uses this estimated uncertainty to determine a mixture ratio for combining actual and predicted sensory states of network input. The method is evaluated by conducting a robot learning experiment in which a robot is required to perform a sensory-dependent task and a sensory-independent task. The sensory-dependent task requires the robot to incorporate meaningful sensory information, and the sensory-independent task requires the robot to ignore irrelevant sensory information. Experimental results demonstrate that a robot controlled by our proposed method exhibits flexible and robust behavior, which results from dynamic modulation of the network input on the basis of the estimated uncertainty of actual sensory states.

We suggest that different behavior generation schemes, such as sensory reflex behavior and intentional proactive behavior, can be developed by a newly proposed dynamic neural network model, named stochastic multiple timescale recurrent neural network (S-MTRNN). The model learns to predict subsequent sensory inputs, generating both their means and their uncertainty levels in terms of variance (or inverse precision) by utilizing its multiple timescale property. This model was employed in robotics learning experiments in which one robot controlled by the S-MTRNN was required to interact with another robot under the condition of uncertainty about the other's behavior. The experimental results show that self-organized and sensory reflex behavior-based on probabilistic prediction-emerges when learning proceeds without a precise specification of initial conditions. In contrast, intentional proactive behavior with deterministic predictions emerges when precise initial conditions are available. The results also showed that, in situations where unanticipated behavior of the other robot was perceived, the behavioral context was revised adequately by adaptation of the internal neural dynamics to respond to sensory inputs during sensory reflex behavior generation. On the other hand, during intentional proactive behavior generation, an error regression scheme by which the internal neural activity was modified in the direction of minimizing prediction errors was needed for adequately revising the behavioral context. These results indicate that two different ways of treating uncertainty about perceptual events in learning, namely, probabilistic modeling and deterministic modeling, contribute to the development of different dynamic neuronal structures governing the two types of behavior generation schemes.

Humans can learn a language through physical interaction with their environment and semiotic communication with other people. It is very important to obtain a computational understanding of how humans can form symbol systems and obtain semiotic skills through their autonomous mental development. Recently, many studies have been conducted regarding the construction of robotic systems and machine learning methods that can learn a language through embodied multimodal interaction with their environment and other systems. Understanding human?-social interactions and developing a robot that can smoothly communicate with human users in the long term require an understanding of the dynamics of symbol systems. The embodied cognition and social interaction of participants gradually alter a symbol system in a constructive manner. In this paper, we introduce a field of research called symbol emergence in robotics (SER). SER represents a constructive approach towards a symbol emergence system. The symbol emergence system is socially self-organized through both semiotic communications and physical interactions with autonomous cognitive developmental agents, i.e. humans and developmental robots. In this paper, specifically, we describe some state-of-art research topics concerning SER, such as multimodal categorization, word discovery, and double articulation analysis. They enable robots to discover words and their embodied meanings from raw sensory-motor information, including visual information, haptic information, auditory information, and acoustic speech signals, in a totally unsupervised manner. Finally, we suggest future directions for research in SER.

In this paper, we propose a model that can explain the mechanism of self and non-self discrimination. Infants gradually develop their abilities for self–other cognition through interaction with the environment. Predictive learning has been widely used to explain the mechanism of infants’ development. We hypothesized that infants’ cognitive abilities are developed through predictive learning and the uncertainty estimation of their sensory-motor inputs. We chose a stochastic continuous time recurrent neural network, which is a dynamical neural network model, to predict uncertainties as variances. From the perspective of cognitive developmental robotics, a predictive learning experiment with a robot was performed. The results indicate that training made the robot predict the regions related to its body more easily. We confirmed that self and non-self cognitive abilities might be acquired through predictive learning with uncertainty estimation.

Content-Based Image Retrieval (CBIR) system enables us to access images using only images as queries, instead of keywords. Photorealistic images, and hand-drawn sketch image can be used as a queries as well. Recently, convolutional neural networks (CNNs) are used to discriminate images including sketches. However, the tasks are limited to classifying only one type of images, either photo or sketch images, due to the lack of a large dataset of sketch images and the large difference of their visual characteristics. In this paper, we introduce a simple way to prepare training datasets, which can enable the CNN model to classify both types of images by color transforming photo and illustration images. Through the training experiment, we show that the proposed method contributes to the improvement of classification accuracy.

We propose a method of tool use considering the transition process of a body model from not grasping to grasping a tool using a single model. In our previous research, we proposed a tool-body assimilation model in which a robot autonomously learns tool functions using a deep neural network (DNN) and recurrent neural network (RNN) through experiences of motor babbling. However, the robot started its motion already holding the tools. In real-life situations, the robot would make decisions regarding grasping (handling) or not grasping (manipulating) a tool. To achieve this, the robot performs motor babbling without the tool pre-attached to the hand with the same motion twice, in which the robot handles the tool or manipulates without graping it. To evaluate the model, we have the robot generate motions with showing the initial and target states. As a result, the robot could generate the correct motions with grasping decisions.

Meanings of language expressions are constructed not only from words grounded in real-world matters, but also from words such as “not” that participate in the construction by working as logical operators. This study proposes a connectionist method for learning and internally representing functions that deal with both of these word groups, and grounding sentences constructed from them in corresponding behaviors just by experiencing raw sequential data of an imposed task. In the experiment, a robot implemented with a recurrent neural network is required to ground imperative positive and negative sentences given as a sequence of words in corresponding goal-oriented behavior. Analysis of the internal representations reveals that the network fulfilled the requirement by extracting XOR problems implicitly included in the target sequences and solving them by learning to represent the logical operations in its nonlinear dynamics in a self-organizing manner.

Drawing is a way of visually expressing our feelings, knowledge, and situation. People draw pictures to share information with other human beings. This study investigates visuomotor memory (VM), which is a reusable memory storing drawing behavioral data. We propose a neural network-based model for acquiring a computational memory that can replicate VM through self-organized learning of a robot's actual drawing experiences. To design the model, we assume that VM has the following two characteristics: (1) it is formed by bottom-up learning and integration of temporal drawn pictures and motion data, and (2) it allows the observers to associate drawing motions from pictures. The proposed model comprises a deep neural network for dimensionally compressing temporal drawn images and a continuous-time recurrent neural network for integration learning of drawing motions and temporal drawn images. Two experiments are conducted on unicursal shape learning to investigate whether the proposed model can learn the function without any shape information for visual processing. Based on the first experiment, the model can learn 15 drawing sequences for three types of pictures, acquiring associative memory for drawing motions through the bottom-up learning process. Thus, it can associate drawing motions from untrained drawn images. In the second experiment, four types of pictures are trained, with four distorted variations per type. In this case, the model can organize the different shapes based on their distortions by utilizing both the image information and the drawing motions, even if visual characteristics are not shared. (C) 2016 The Authors. Published by Elsevier B.V.

Noise robust speech recognition is crucial for effective human-machine interaction in real-world environments. Sound source separation (SSS) is one of the most widely used approaches for addressing noise robust speech recognition by extracting a target speaker's speech signal while suppressing simultaneous unintended signals. However, conventional SSS algorithms, such as independent component analysis or nonlinear principal component analysis, are limited in modeling complex projections with scalability. Moreover, conventional systems required designing an independent subsystem for noise reduction (NR) in addition to the SSS. To overcome these issues, we propose a deep neural network (DNN) framework for modeling the separation function (SF) of an SSS system. By training a DNN to predict clean sound features of a target sound from corresponding multichannel deteriorated sound feature inputs, we enable the DNN to model the SF for extracting the target sound without prior knowledge regarding the acoustic properties of the surrounding environment. Moreover, the same DNN is trained to function simultaneously as a NR filter. Our proposed SSS system is evaluated using an isolated word recognition task and a large vocabulary continuous speech recognition task when either nondirectional or directional noise is accumulated in the target speech. Our evaluation results demonstrate that DNN performs noticeably better than the baseline approach, especially when directional noise is accumulated with a low signal-to-noise ratio.

We propose a method for realizing effective dynamic motion learning in a flexible-joint robot using motor babbling. Flexible-joint robots have recently attracted attention because of their adaptiveness, safety, and, in particular, dynamic motions. It is difficult to control robots that require dynamic motion. In past studies, attractors and oscillators were designed as motion primitives of an assumed task in advance. However, it is difficult to adapt to unintended environmental changes using such methods. To overcome this problem, we use a recurrent neural network (RNN) that does not require predetermined parameters. In this research, we propose a method for facilitating effective learning. First, a robot learns simple motions via motor babbling, acquiring body dynamics using a recurrent neural network (RNN). Motor babbling is the process of movement that infants use to acquire their own body dynamics during their early days. Next, the robot learns additional motions required for a target task using the acquired body dynamics. For acquiring these body dynamics, the robot uses motor babbling with its redundant flexible joints to learn motion primitives. This redundancy implies that there are numerous possible motion patterns. In comparison to a basic learning task, the motion primitives are simply modified to adjust to the task. Next, we focus on the types of motions used in motor babbling. We classify the motions into two motion types, passive motion and active motion. Passive motion involves inertia without any torque input, whereas active motion involves a torque input. The robot acquires body dynamics from the passive motion and a means of torque generation from the active motion. As a result, we demonstrate the importance of performing prior learning via motor babbling before learning a task. In addition, task learning is made more efficient by dividing the motion into two types of motor babbling patterns.

In this study, we propose a neural network based model for learning a robot's drawing sequences in an unsupervised manner. We focus on the ability to learn visual-motor relationships, which can work as a reusable memory in association of drawing motion from a picture image. Assuming that a humanoid robot can draw a shape on a pen tablet, the proposed model learns drawing sequences, which comprises drawing motion and drawn picture image frames. To learn raw pixel data without any given specific features, we utilized a deep neural network for compressing large dimensional picture images and a continuous time recurrent neural network for integration of motion and picture images. To confirm the ability of the proposed model, we performed an experiment for learning 15 sequences comprising three types of shapes. The model successfully learns all the sequences and can associate a drawing motion from a not trained picture image and a trained picture with similar success. We also show that the proposed model self-organizes its behavior according to types shapes.

In recent years there has been increased interest in studies that explore integrative learning of language and other modalities by using neural network models. However, for practical application to human-robot interaction, the acquired semantic structure between language and meaning has to be available immediately and repeatably whenever necessary, just as in everyday communication. As a solution to this problem, this study proposes a method in which a recurrent neural network self-organizes cyclic attractors that reflect semantic structure and represent interaction flows in its internal dynamics. To evaluate this method we design a simple task in which a human verbally directs a robot, which responds appropriately. Training the network with training data that represent the interaction series, the cyclic attractors that reflect the semantic structure is self-organized. The network first receives a verbal direction, and its internal state moves according to the first half of the cyclic attractors with branch structures corresponding to semantics. After that, the internal state reaches a potential to generate appropriate behavior. Finally, the internal state moves to the second half and converges on the initial point of the cycle while generating the appropriate behavior. By self-organizing such an internal structure in its forward dynamics, the model achieves immediate and repeatable response to linguistic directions. Furthermore, the network self-organizes a fixed-point attractor, and so able to wait for directions. It can thus repeat the interaction flexibly without explicit turn-taking signs.

This paper introduces an imitative model that enables a robot to acquire viewpoints of the self and others from its own sensory-motor experiences. This is important for recognizing and imitating actions generated from various directions. Existing methods require coordinate transformations input by human designers or complex learning modules to acquire a viewpoint. In the proposed model, several neurons dedicated to generated actions and viewpoints of the self and others are added to a dynamic nueral network model reffered as continuous time recurrent neural network (CTRNN). The training data are labeled with types of actions and viewpoints, and are linked to each internal state. We implemented this model in a robot and trained the model to perform actions of object manipulation. Representations of behavior and viewpoint were formed in the internal states of the CTRNN. In addition, we analyzed the initial values of the internal states that represent the viewpoint information. We confirmed the distinction of the observational perspective of other's actions self-organized in the space of the initial values. Combining the initial values of the internal states that describe the behavior and the viewpoint, the system can generate unlearned data.

Dynamic interactions with caregivers are essential for infants to develop cognitive abilities, including aspects of action, perception, and attention. We hypothesized that these abilities can be acquired through the predictive learning of sensory inputs including their uncertainty (inverse precision) in terms of variance. To examine our hypothesis from the perspective of cognitive developmental robotics, we conducted a neurorobotics experiment involving a ball-playing interaction task between a human experimenter representing a caregiver and a small humanoid robot representing an infant. The robot was equipped with a dynamic generative model called a stochastic continuous-time recurrent neural network (S-CTRNN). The S-CTRNN learned to generate predictions about both the visuo-proprioceptive states of the robot and the uncertainty of these states by minimizing a negative log-likelihood consisting of log-uncertainty and precision-weighted prediction error. The experimental results showed that predictive learning with uncertainty estimation enabled the robot to acquire infant-like cognitive abilities through dynamic interactions with the experimenter. We also discuss the effects of infant-directed modifications observed in caregiver-infant interactions on the development of these abilities.

Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for reliable speech recognition, particularly when the audio is corrupted by noise. However, cautious selection of sensory features is crucial for attaining high recognition performance. In the machine-learning community, deep learning approaches have recently attracted increasing attention because deep neural networks can effectively extract robust latent features that enable various recognition algorithms to demonstrate revolutionary generalization capabilities under diverse application conditions. This study introduces a connectionist-hidden Markov model (HMM) system for noise-robust AVSR. First, a deep denoising autoencoder is utilized for acquiring noise-robust audio features. By preparing the training data for the network with pairs of consecutive multiple steps of deteriorated audio features and the corresponding clean features, the network is trained to output denoised audio features from the corresponding features deteriorated by noise. Second, a convolutional neural network (CNN) is utilized to extract visual features from raw mouth area images. By preparing the training data for the CNN as pairs of raw images and the corresponding phoneme label outputs, the network is trained to predict phoneme labels from the corresponding mouth area input images. Finally, a multi-stream HMM (MSHMM) is applied for integrating the acquired audio and visual HMMs independently trained with the respective features. By comparing the cases when normal and denoised mel-frequency cepstral coefficients (MFCCs) are utilized as audio features to the HMM, our unimodal isolated word recognition results demonstrate that approximately 65 % word recognition rate gain is attained with denoised MFCCs under 10 dB signal-to-noise-ratio (SNR) for the audio signal input. Moreover, our multimodal isolated word recognition results utilizing MSHMM with denoised MFCCs and acquired visual features demonstrate that an additional word recognition rate gain is attained for the SNR conditions below 10 dB.

We propose an exploratory motor babbling utilizing variance prediction of recurrent neural network as a method to explore body dynamics of robot with flexible joints. With conventional research methods, practical use with real robots is difficult because of the large numbers of required motor babbling motions. With the proposed method, over-fitting of predictable motions is reduced by sequentially learning appropriate motion for body model from unpredictable motions. To evaluate the method, an experiment where the robot additionally learns crank turning task after the exploration of body dynamics were conducted. The results show that the proposed method is capable of efficient motion generation in any given motion task.

We are currently developing an open-framework for RT-middleware which advances the reusability of RT-component and RT-system. Using the RT-middleware, though the softwares of the robotics elements like actuators, controllers, and sensors, are encapsulated into RT-component which can be easily reused, the way how to collect the source-codes of RTCs is not well-discussed yet. In our open-framework, both a repository management and a build support system are implemented. In this paper, the RTC source-code build-status management system which uses both the repository management and the build support system is shown.

In this paper, we propose a system framework of a mapping and navigation robot which does not depend on specific robot middleware platforms. Recently, robot middleware have attracted attention widely because that is expected to reduce the cost of robot development. However, many robot middleware platforms have been developed, so it must to define a common framework to design robot system without depending on specific platform. Therefore, we propose an autonomous mobile robot framework which independent on specific robot middleware platforms, hi addition, this model are implemented in RT-Middleware and experimentally evaluated.

Intrinsic motivation is one of the keys in implementing the mechanism of interest to robots. In this paper, we present a method to apply intrinsic motivation in dynamics learning with predictable and unpredictable targets in view. The robots arm is used for the predictable target and the humans arm is used for the unpredictable target in the experiment. The learning algorithm based on intrinsic motivation will automatically set a larger weight to targets that would contribute to decreasing the training error, while setting a smaller weight to others. A neurodynamical model, namely multiple timescales recurrent neural network (MTRNN), is utilized for studying the robots arm/external object dynamics. Training of MTRNN is done using the back propagation through time (BPTT) algorithm. We modify the BPTT algorithm by the following two steps. (1) Evaluate predictability of robots arm/objects using training error of MTRNN. (2) Assign a preference ratio, which represents the weight of the training, to each object based on predictability. The proposed training method would focus more on reducing training error of predictable objects compared to normal BPTT, where training error is equally treated for every object. Experiments were conducted using an actual robot platform, moving the robots arm while a human moves his arm in the robots camera view. The results of the experiment showed that the proposed training method could achieve smaller training error of the robots arm visuomotor dynamics, which is predictable from the robots motor command, compared to general training with BPTT. Evaluation of MTRNN as a forward model to predict untrained data showed that the proposed model is capable of predicting the robots hand motion, specifically with larger number of nodes.

We propose the new method of tool use with a tool-body assimilation model based on body babbling and a neurodynamical system for robots to use tools. Almost all existing studies for robots to use tools require predetermined motions and tool features; the motion patterns are limited and the robots cannot use novel tools. Other studies fully search for all available parameters for novel tools, but this leads to massive amounts of calculations. To solve these problems, we took the following approach: we used a humanoid robot model to generate random motions based on human body babbling. These rich motion experiences were used to train recurrent and deep neural networks for modeling a body image. Tool features were self-organized in parametric bias, modulating the body image according to the tool in use. Finally, we designed a neural network for the robot to generate motion only from the target image. Experiments were conducted with multiple tools for manipulating a cylindrical target object. The results show that the tool-body assimilation model is capable of motion generation.

Recognizing grasped objects with tactile sensors is beneficial in many situations, as other sensor information like vision is not always reliable. In this paper, we aim for multimodal object recognition by power grasping of objects with an unknown orientation and position relation to the hand. Few robots have the necessary tactile sensors to reliably recognize objects: in this study the multifingered hand of TWENDY-ONE is used, which has distributed skin sensors covering most of the hand, 6 axis F/T sensors in each fingertip, and provides information about the joint angles. Moreover, the hand is compliant. When using tactile sensors, it is not clear what kinds of features are useful for object recognition. Recently, deep learning has shown promising results. Nevertheless, deep learning has rarely been used in robotics and to our best knowledge never for tactile sensing, probably because it is difficult to gather many samples with tactile sensors. Our results show a clear improvement when using a denoising autoencoder with dropout compared to traditional neural networks. Nevertheless, a higher number of layers did not prove to be beneficial.

We propose an exploratory form of motor babbling that uses variance predictions from a recurrent neural network as a method to acquire the body dynamics of a robot with flexible joints. In conventional research methods, it is difficult to construct real robots because of the large number of motor babbling motions required. In motor babbling, different motions may be easy or difficult to predict. The variance is large in difficult-to-predict motions, whereas the variance is small in easy-topredict motions. We use a Stochastic Continuous Timescale Recurrent Neural Network to predict the accuracy and variance of motions. Using the proposed method, a robot can explore motions based on variance. To evaluate the proposed method, experiments were conducted in which the robot learns crank turning and door opening/closing tasks after exploring its body dynamics. The results show that the proposed method is capable of efficient motion generation for any given motion tasks.

This paper investigates the essential difference between two types of behavior generation schemes, namely, sensory reflex behavior generation and intentional proactive behavior generation, by proposing a dynamic neural network model referred to as stochastic multiple-timescale recurrent neural network (S-MTRNN). The proposed model was employed in an experiment involving robots learning to cooperate with others under the condition of potential unpredictability of the others' behaviors. The results of the learning experiment showed that sensory reflex behavior was generated by a self-organizing probabilistic prediction mechanism when the initial sensitivity characteristics in the network dynamics were not utilized in the learning process. In contrast, proactive behavior with a deterministic prediction mechanism was developed when the initial sensitivity was utilized. It was further shown that in situations where unexpected behaviors of others were observed, the behavioral context was re-situated by adaptation of the internal neural dynamics by means of simple sensory reflexes in the former case. In the latter case, the behavioral context was re-situated by error regression of the internal neural activity rather than by sensory reflex. The role of the top-down and bottom-up interactions in dealing with unexpected situations is discussed.

This paper discusses a possible neurodynamic mechanism that enables self-organization of two basic behavioral modes, namely a proactive mode and a reactive mode, and of autonomous switching between these modes depending on the situation. In the proactive mode, actions are generated based on an internal prediction, whereas in the reactive mode actions are generated in response to sensory inputs in unpredictable situations. In order to investigate how these two behavioral modes can be self-organized and how autonomous switching between the two modes can be achieved, we conducted neurorobotics experiments by using our recently developed dynamic neural network model that has a capability to learn to predict time-varying variance of the observable variables. In a set of robot experiments under various conditions, the robot was required to imitate others movements consisting of alternating predictable and unpredictable patterns. The experimental results showed that the robot controlled by the neural network model was able to proactively imitate predictable patterns and reactively follow unpredictable patterns by autonomously switching its behavioral modes. Our analysis revealed that variance prediction mechanism can lead to self-organization of these abilities with sufficient robustness and generalization capabilities. © 2014 © 2014 Taylor & Francis and The Robotics Society of Japan.

For humans to accurately understand the world around them, multimodal integration is essential because it enhances perceptual precision and reduces ambiguity. Computational models replicating such human ability may contribute to the practical use of robots in daily human living environments; however, primarily because of scalability problems that conventional machine learning algorithms suffer from, sensory-motor information processing in robotic applications has typically been achieved via modal-dependent processes. In this paper, we propose a novel computational framework enabling the integration of sensory-motor time-series data and the self-organization of multimodal fused representations based on a deep learning approach. To evaluate our proposed model, we conducted two behavior-learning experiments utilizing a humanoid robot; the experiments consisted of object manipulation and bell-ringing tasks. From our experimental results, we show that large amounts of sensory-motor information, including raw RGB images, sound spectrums, and joint angles, are directly fused to generate higher-level multimodal representations. Further, we demonstrated that our proposed framework realizes the following three functions: (1) cross-modal memory retrieval utilizing the information complementation capability of the deep autoencoder; (2) noise-robust behavior recognition utilizing the generalization capability of multimodal features; and (3) multimodal causality acquisition and sensory-motor prediction based on the acquired causality. © 2014 Elsevier B.V. All rights reserved.

神経回路モデル を用いた道具身体化と動作の獲得を提案する.従来の道具使用モデルでは，道具の特徴量，身体モデル，動作を設計する必要があった．我々は，事前設計を必要としない手法を神経回路モデルで行う．道具の特徴量は，Self Organized Mapを用いて画像から自己組織的に獲得する．動作については，人間の発達過程にみられる身体バブリングを参考に学習させる．具体的には，ランダム動作中の感覚運動データから，神経回路モデルMultipleTime-scales Neural Network(MTRNN)に自己の身体モデルを獲得させる．道具使用時は学習済みのMTRNNにParametric Biasを付加することで獲得した身体モデルを変更することなく動作生成を実現する．実験はシミュレーション 上で, 素手による物体操作で身体モデルの学習後，T字型の道具による物体操作を行った.結果として, 目標画像を与えることでその目標状態に近くなるように引き寄せ動作を生成することが確認できた．

In this paper, we propose a tool-body assimilation model that implements a multiple time-scales recurrent neural network (MTRNN). Our model allows a robot to acquire the representation of a tool function and the required motion without having any prior knowledge of the tool. It is composed of five modules: image feature extraction, body model, tool dynamics feature, tool recognition, and motion recognition. Self-organizing maps (SOM) are used for image feature extraction from raw images. The MTRNN is used for body model learning. Parametric bias (PB) nodes are used to learn tool dynamic features. The PB nodes are attached to the neurons of the MTRNN to modulate the body model. A hierarchical neural network (HNN) is implemented for tool and motion recognition. Experiments were conducted using OpenHRP3, a robotics simulator, with multiple tools. The results show that the tool-body assimilation model is capable of recognizing tools, including those having an unlearned shape, and acquires the required motions accordingly. © 2014 IEEE.

In this paper, we present a method to apply intrinsic motivation for improving visuomotor learning of robot's arm with external object in view. Multiple Timescales Recurrent Neural Network (MTRNN) is utilized for learning the robot arm/external object dynamics. Training of MTRNN is done using the Back Propagation Through Time (BPTT) algorithm. BPTT algorithm is modified as follows. 1. Evaluate predictability of robot arm/objects using training error of MTRNN. 2. Assign a preference ratio to each object based on predictability. The preference ratio represents the weight of each object to training. Experiments were conducted using an actual robot moving the arm while a human moves his arm in the robot's camera view. The result of the experiment showed that the proposed method presents better training result of robot arm visuomotor dynamics compared to general training with BPTT.

In recent automatic speech recognition studies, deep learning architecture applications for acoustic modeling have eclipsed conventional sound features such as Mel-frequency cepstral co- efficients. However, for visual speech recognition (VSR) stud- ies, handcrafted visual feature extraction mechanisms are still widely utilized. In this paper, we propose to apply a convo- lutional neural network (CNN) as a visual feature extraction mechanism for VSR. By training a CNN with images of a speaker's mouth area in combination with phoneme labels, the CNN acquires multiple convolutional filters, used to extract vi- sual features essential for recognizing phonemes. Further, by modeling the temporal dependencies of the generated phoneme label sequences, a hidden Markov model in our proposed sys- Tem recognizes multiple isolated words. Our proposed system is evaluated on an audio-visual speech dataset comprising 300 Japanese words with six different speakers. The evaluation re- sults of our isolated word recognition experiment demonstrate that the visual features acquired by the CNN significantly out- perform those acquired by conventional dimensionality com- pression approaches, including principal component analysis.

In this paper, we propose a tool-body assimilation model that implements a multiple time-scales recurrent neural network (MTRNN). Our model allows a robot to acquire the representation of a tool function and the required motion without having any prior knowledge of the tool. It is composed of five modules: image feature extraction, body model, tool dynamics feature, tool recognition, and motion recognition. Self-organizing maps (SOM) are used for image feature extraction from raw images. The MTRNN is used for body model learning. Parametric bias (PB) nodes are used to learn tool dynamic features. The PB nodes are attached to the neurons of the MTRNN to modulate the body model. A hierarchical neural network (HNN) is implemented for tool and motion recognition. Experiments were conducted using OpenHRP3, a robotics simulator, with multiple tools. The results show that the tool-body assimilation model is capable of recognizing tools, including those having an unlearned shape, and acquires the required motions accordingly. © 2014 IEEE.

We aim to achieve interaction between a robot and multiple people. For this, robots should localize people, select an interaction partner, and act appropriately for him/her. It is difficult to deal with all these problems using only the sensors installed into the robots. We focus on that people use a rough interaction distance among other people . We divide this interaction area into different spaces based on both the interaction distances and sensor abilities of robots. Our robots localize people roughly within this divided space. To select an interaction partner, they map friendliness holding the interaction history onto the divided space, and integrate the sensor information. Furthermore, we developed a method for appropriately changing the motions, sizes, and speeds based on the distance. Our robots regard the divided spaces as Q-Learning states, and learn the motion parameters. Our robot interacted with 27 visitors. It localized a partner with an F-value of 0.76 through integration, which is higher than that of a single sensor. A factor analysis was performed on the results from questionnaires. Exciting and Friendly were the representatives of the first and second factors, respectively. For both factors, a motion with friendliness provided higher impression scores than that without friendliness. © 2013 Taylor & Francis and The Robotics Society of Japan.

We propose a model for robots to use tools without predetermined parameters based on a human cognitive model. Almost all existing studies of robot using tool require predetermined motions and tool features, so the motion patterns are limited and the robots cannot use new tools. Other studies use a full search for new tools; however, this entails an enormous number of calculations. We built a model for tool use based on the phenomenon of tool-body assimilation using the following approach: We used a humanoid robot model to generate random motion, based on human body babbling. These rich motion experiences were then used to train a recurrent neural network for modeling a body image. Tool features were self-organized in the parametric bias modulating the body image according to the used tool. Finally, we designed the neural network for the robot to generate motion only from the target image. © 2014 Springer International Publishing Switzerland.

In the present study, we demonstrate the learning and recognition capabilities of our recently proposed recurrent neural network (RNN) model called stochastic continuous-time RNN (S-CTRNN). S-CTRNN can learn to predict not only the mean but also the variance of the next state of the learning targets. The network parameters consisting of weights, biases, and initial states of context neurons are optimized through maximum likelihood estimation (MLE) using the gradient descent method. First, we clarify the essential difference between the learning capabilities of conventional CTRNN and S-CTRNN by analyzing the results of a numerical experiment in which multiple fluctuating temporal patterns were used as training data, where the variance of the Gaussian noise varied among the patterns. Furthermore, we also show that the trained S-CTRNN can recognize given fluctuating patterns by inferring the initial states that can reproduce the patterns through the same MLE scheme as that used for network training. © 2014 Springer International Publishing Switzerland.

This paper proposes an approach for robots to ac-quire language grounding in their robot's sensory-motor flow using neuro-dynamical models. We trained our neuro-dynamical model over a set of sentences represented as sequences of characters. For the integrated recognition, we introduced a cognitive hypothesis for integrated recognition where a human's brain separately processed the 'structure' and 'contents' of a sentence. Our model was trained with the spelling of words and their semantic role emerged in the first model. As a result of binding the model with sensory-motion patterns, we confirmed that it could associate proper word spellings with a sensory-motor flows and a semantic roles, even if an observed flow had not been learned. © 2013 IEEE.

We present a Bayesian analysis method that estimates the harmonic structure of musical instruments in music signals on the basis of psychoacoustic evidence. Since the main objective of multipitch analysis is joint estimation of the fundamental frequencies and their harmonic structures, the performance of harmonic structure estimation significantly affects fundamental frequency estimation accuracy. Many methods have been proposed for estimating the harmonic structure accurately, but no method has been proposed that satisfies all these requirements: robust against initialization, optimization-free, and psychoacoustically appropriate and thus easy to develop further. Our method satisfies these requirements by explicitly incorporating Terhardt's virtual pitch theory within a Bayesian framework. It does this by automatically learning the valid weight range of the harmonic components using a MIDI synthesizer. The bounds are termed "overtone corpus." Modeling demonstrated that the proposed overtone corpus method can stably estimate the harmonic structure of 40 musical pieces for a wide variety of initial settings. © 2013 Information Processing Society of Japan.

This paper mainly deals with robot developmental learning on drawing and discusses the influences of physical embodiment to the task. Humans are said to develop their drawing skills through five phases: 1) Scribbling, 2) Fortuitous Realism, 3) Failed Realism, 4) Intellectual Realism, 5) Visual Realism. We implement phases 1) and 3) into the humanoid robot NAO, holding a pen, using a neuro dynamical model, namely Multiple Timescales Recurrent Neural Network (MTRNN). For phase 1), we used random arm motion of the robot as body babbling to associate motor dynamics with pen position dynamics. For phase 3), we developed incremental imitation learning to imitate and develop the robot's drawing skill using basic shapes: circle, triangle, and rectangle. We confirmed two notable features from the experiment. First, the drawing was better performed for shapes requiring arm motions used in babbling. Second, performance of clockwise drawing of circle was good from beginning, which is a similar phenomenon that can be observed in human development. The results imply the capability of the model to create a developmental robot relating to human development. © 2013 IEEE.

Our brain is known to enhance perceptual precision and reduce ambiguity about sensory environment by integrating multiple sources of sensory information acquired from different modalities, such as vision, auditory and somatic sensation. From an engineering perspective, building a computational model that replicates this ability to integrate multimodal information and to self-organize the causal dependency among them, represents one of the central challenges in robotics. In this study, we propose such a model based on a deep learning framework and we evaluate the proposed model by conducting a bell ring task using a small humanoid robot. Our experimental results demonstrate that (1) the cross-modal memory retrieval function of the proposed method succeeds in generating visual sequence from the corresponding sound and bell ring motion, and (2) the proposed method leads to accurate causal dependencies among the sensory-motor sequence. © 2013 IEEE.

This paper presents a novel computational approach for modeling and generating multiple object manipulation behaviors by a humanoid robot. The contribution of this paper is that deep learning methods are applied not only for multimodal sensor fusion but also for sensory-motor coordination. More specifically, a time-delay deep neural network is applied for modeling multiple behavior patterns represented with multi-dimensional visuomotor temporal sequences. By using the efficient training performance of Hessian-free optimization, the proposed mechanism successfully models six different object manipulation behaviors in a single network. The generalization capability of the learning mechanism enables the acquired model to perform the functions of cross-modal memory retrieval and temporal sequence prediction. The experimental results show that the motion patterns for object manipulation behaviors are successfully generated from the corresponding image sequence, and vice versa. Moreover, the temporal sequence prediction enables the robot to interactively switch multiple behaviors in accordance with changes in the displayed objects. © 2013 IEEE.

Robots are required to process multimodal information because the information in the real world comes from various modal inputs. However, there exist only a few robots integrating multimodal information. Humans can recognize the environment effectively by cross-modal processing. We focus on modeling synaesthesia phenomenon known to be a cross-modal perception of humans. Recently, deep neural networks (DNNs) have gained more attention and successfully applied to process high-dimensional data composed not only of single modality but also of multimodal information. We introduced DNNs to construct multimodal association model which can reconstruct one modality from the other modality. Our model is composed of two DNNs: one for image compression and the other for audio-visual sequential learning. We tried to reproduce synaesthesia phenomenon by training our model with the multimodal data acquired from psychological experiment. Cross-modal association experiment showed that our model can reconstruct the same or similar images from sound as synaesthetes, those who experience synaesthesia. The analysis of middle layers of DNNs representing multimodal features implied that DNNs self-organized the difference of perception between individual synaesthetes. © 2013 IEEE.

The optimal way to build speech understanding modules depends on the amount of training data available. When only a small amount of training data is available, effective allocation of the data is crucial to preventing overfitting of statistical methods. We have developed a method for allocating a limited amount of training data in accordance with the amount available. Our method exploits rule-based methods for when the amount of data is small, which are included in our speech understanding framework based on multiple model combinations, i.e., multiple automatic speech recognition (ASR) modules and multiple language understanding (LU) modules, and then allocates training data preferentially to the modules that dominate the overall performance of speech understanding. Experimental evaluation showed that our allocation method consistently outperforms baseline methods that use a single ASR module and a single LU module while the amount of training data increases. Copyright © 2012 The Institute of Electronics, Information and Communication Engineers.

We present a method to recuperate fingerings for a given piece of violin music in order to recreate the timbre of a given audio recording of the piece. This is achieved by first analyzing an audio signal to determine the most likely sequence of two-dimensional fingerboard locations (string number and location along the string), which recovers elements of violin fingering relevant to timbre. This sequence is then used as a constraint for finding an ergonomic sequence of finger placements that satisfies both the sequence of notated pitch and the given fingerboard-location sequence. Fingerboard-location-sequence estimation is based on estimation of a hidden Markov model, each state of which represents a particular fingerboard location and emits a Gaussian mixture model of the relative strengths of harmonics. The relative strengths of harmonics are estimated from a polyphonic mixture using score-informed source segregation, and compensates for discrepancies between observed data and training data through mean normalization. Fingering estimation is based on the modeling of a cost function for a sequence of finger placements. We tailor our model to incorporate the playing practices of the violin. We evaluate the performance of the fingerboard-location estimator with a polyphonic mixture, and with recordings of a violin whose timbral characteristics differ significantly from that of the training data. We subjectively evaluate the fingering estimator and validate the effectiveness of tailoring the fingering model towards the violin. © 2012 Massachusetts Institute of Technology.

Our motivation is to develop a robot that treats auditory information in real environment because auditory information is useful for animated communications or understanding our surroundings. Interactions by using sound information need an aquisition of it and a proper sound source reference between a user and a robot leads to it. Such sound source reference is difficult due to multiple sound sources generating in real environemnt, and we use onomatopoeic representations as a representation for the reference. This paper shows a system that selects a sound source specified by a user from multiple sound sources. Users use onomatopoeias in the specification, and our system separates a mixed sound and converts separated sounds into onomatopoeias for the selection. Onomatopoeais have the ambiguity that each user gives each expression to a certain sound and we create an original similarity based on Minimum Edit Distance and acoustic features for solving its problem. In experiments, our system receives a mixed sound consisting of three sounds and a user's query as inputs, and checks a count of a consistency of a sound source selected by a system and a sound source specified by a user in 100 tests. The result shows © 2012 IEEE.

Researches in the brain science field have uncovered the human capability to use tools as if they are part of the human bodies (known as tool-body assimilation) through trial and experience. This paper presents a method to apply a robot's active sensing experience to create the tool-body assimilation model. The model is composed of a feature extraction module, dynamics learning module, and a tool-body assimilation module. Self-organizing map (SOM) is used for the feature extraction module to extract object features from raw images. Multiple time-scales recurrent neural network (MTRNN) is used as the dynamics learning module. Parametric bias (PB) nodes are attached to the weights of MTRNN as second-order network to modulate the behavior of MTRNN based on the properties of the tool. The generalization capability of neural networks provide the model the ability to deal with unknown tools. Experiments were conducted with the humanoid robot HRP-2 using no tool, I-shaped, T-shaped, and L-shaped tools. The distribution of PB values have shown that the model has learned that the robot's dynamic properties change when holding a tool. Motion generation experiments show that the tool-body assimilation model is capable of applying to unknown tools to generate goal-oriented motions. © 2012 IEEE.

This letter presents a new algorithm for blind dereverberation and echo cancellation based on independent component analysis (ICA) for actual acoustic signals. We focus on frequency domain ICA (FD-ICA) because its computational cost and speed of learning convergence are sufficiently reasonable for practical applications such as hands-free speech recognition. In applying conventional FD-ICA as a preprocessing of automatic speech recognition in noisy environments, one of the most critical problems is how to copewith reverberations. To extract a clean signal from the reverberant observation, we model the separation process in the shorttime Fourier transform domain and apply the multiple input/output inverse-filtering theorem (MINT) to the FD-ICA separation model. A naive implementation of this method is computationally expensive, because its time complexity is the second order of reverberation time. Therefore, themain issue in dereverberation is to reduce the high computational cost of ICA. In this letter, wereduce the computational complexity to the linear order of the reverberation time by using two techniques: (1) a separation model based on the independence of delayed observed signals with MINT and (2) spatial sphering for preprocessing. Experiments show that the computational cost grows in proportion to the linear order of the reverberation time and that ourmethod improves the word correctness of automatic speech recognition by 10 to 20 points in a RT20 =670 ms reverberant environment. © 2011 Massachusetts Institute of Technology.

Music has long been used to strengthen bonds between humans. In our research, we develop musical coplayer robots with the hope that music may improve human-robot symbiosis as well. In this paper, we underline the importance of non-verbal, visual communication for ensemble synchronization at the start, during and end of a piece. We propose three cues for interplayer communication, and present a thereminplaying, singing robot that can detect them and adapt its play to a human flutist. Experiments with two naive flutists suggest that the system can recognize naturally occurring flutist gestures without requiring specialized user training. In addition, we show how the use of audio-visual aggregation can allow a robot to adapt to tempo changes quickly. © 2012 Koninklijke Brill NV, Leiden and The Robotics Society of Japan.

The aim of this paper is to improve beat-tracking for live guitar performances. Beat-tracking is a function to estimate musical measurements, for example musical tempo and phase. This method is critical to achieve a synchronized ensemble performance such as musical robot accompaniment. Beat-tracking of a live guitar performance has to deal with three challenges: tempo fluctuation, beat pattern complexity and environmenta noise. To cope with these problems, we devise an audiovisual integration method for beat-tracking. The auditory beat features are estimated in terms of tactus (phase) and tempo (period) by Spectro-Temporal Pattern Matching (STPM), robust against stationary noise. The visual beat features are estimated by tracking the position of the hand relative to the guitar using optical flow, mean shift and the Hough transform. Both estimated features are integrated using a particle filter to aggregate the multimodal information based on a beat location model and a hand's trajectory model. Experimental results confirm that our beat-tracking improves the F-measure by 8.9 points on average over the Murata beat-tracking method, which uses STPM and rule-based beat detection. The results also show that the system is capable of real-time processing with a suppressed number of particles while preserving the estimation accuracy. We demonstrate an ensemble with the humanoid HRP-2 that plays the theremin with a human guitarist. © 2012 Itohara et al; licensee Springer.

It has been long speculated that expression of emotions from different modalities have the same underlying 'code', whether it be a dance step, musical phrase, or tone of voice. This is the first attempt to implement this theory across three modalities, inspired by the polyvalence and repeatability of robotics. We propose a unifying framework to generate emotions across voice, gesture, and music, by representing emotional states as a 4-parameter tuple of speed, intensity, regularity, and extent (SIRE). Our results show that a simple 4-tuple can capture four emotions recognizable at greater than chance across gesture and voice, and at least two emotions across all three modalities. An application for multi-modal, expressive music robots is discussed. © 2012 Lim et al; licensee Springer.

This paper focuses on blind speech separation in under-determined conditions, that is, in the case when there are more sound sources than microphones. We introduce a sound source model based on the Gaussian mixture model (GMM) to represent a speech signal in the time-frequency domain, and derive rules for updating the model parameters using the auxiliary function method. Our GMM sound source model consists of two kinds of Gaussians: sharp ones representing harmonic parts and smooth ones representing nonharmonic parts. Experimental results reveal that our method outperforms the method based on non-negative matrix factorization (NMF) by 0.7dB in the signal-to-distortion ratio (SDR), and by 1.7dB in the signal-to-interference ratio (SIR). This means that our method effectively removes interference coming from other talkers. © 2012 Springer-Verlag.

We present a method of blind source separation (BSS) for speech signals using a complex extension of infinite sparse factor analysis (ISFA) in the frequency domain. Our method is robust against delayed signals that usually occur in real environments, such as reflections, short-time reverberations, and time lags of signals arriving at microphones. ISFA is a conventional non-parametric Bayesian method of BSS, which has only been applied to time domain signals because it can only deal with real signals. Our method uses complex normal distributions to estimate source signals and mixing matrix. Experimental results indicate that our method outperforms the conventional ISFA in the average signal-to-distortion ratio (SDR). © 2012 Springer-Verlag.

We present a new method for modeling the overtone structures of musical instruments that uses an overtone corpus generated using a MIDI synthesizer. Since multipitch estimation requires a joint estimation of F0's and their overtone structures, one of the most important problems is the overtone structure modeling. Latent harmonic allocation (LHA), a promising multipitch estimation method, is difficult to use for various applications because it requires appropriate prior distributions of the overtone structures, which cannot be determined from statistical evidence. Our method uses an overtone corpus to avoid the problem of setting prior distributions and instead restricts the lower and upper bounds of each overtone weight. The bounds are determined from reference signals generated by a MIDI synthesizer. Experimental results demonstrated that the overtone structures were stably and accurately estimated for a wide variety of initial settings. © 2012 IEEE.

Our goal is to give mobile robots a rich representation of their environment as fast as possible. Current mapping methods such as SLAM are often sparse, and scene reconstruction methods using tilting laser scanners are relatively slow. In this paper, we outline a new method for iterative construction of a geometric mesh using streaming time-of-flight range data. Our results show that our algorithm can produce a stable representation after 6 frames, with higher accuracy than raw time-of-flight data. © 2012 IEEE.

This paper proposes a novel hybrid-structured model for the adaptive localization of robots combining a stochastic localization model and a rhythmic action model, for avoiding vacant spaces of landmarks efficiently. In regularly arranged landmark environments, robots may not be able to detect any landmarks for a long time during a straight-like movement. Consequently, locally diverse and smooth movement patterns need to be generated to keep the position estimation stable. Conventional approaches aiming at the probabilistic optimization cannot rapidly generate the detailed movement pattern due to a huge computational cost; therefore a simple but diverse movement structure needs to be introduced as an alternative option. We solve this problem by combining a particle filter as the stochastic localization module and the dynamical action model generating a zig-zagging motion. The validation experiments, where virtual-line-tracing tasks are exhibited on a floor-installed RFID environment, show that introducing the proposed rhythm pattern can improve a minimum error boundary and a velocity performance for arbitrary tolerance errors can be improved by the rhythm amplitude adaptation fed back by the localization deviation. © 2012 IEEE.

We present an adaptive pitch control method for a theremin playing robot in ensemble. The problem of the theremin playing is its sensitivity to the environment. This degrades the pitch accuracy because its pitch characteristics are time varying caused by, such as a co-player motion during the ensemble. We solve this problem using a state space model of this characteristics and an unscented Kalman filter. Experimental results show that our method reduces the pitch error the EKF and block-wise update method by 90% and 77% on average, and the robot can play a musical score of 72.9 cent error on average. © Springer-Verlag Berlin Heidelberg 2012.

We have developed a method that identifies musical chords in polyphonic musical signals. As musical chords mainly represent the harmony of music and are related to other musical elements such as melody and rhythm, we should be able to recognize chords more effectively if this interrelationship is taken into consideration. We use bass pitches as clues for improving chord recognition. The proposed chord recognition system is constructed based on Viterbi-algorithm- based maximum a posteriori estimation that uses a posterior probability based on chord features, chord transition patterns, and bass pitch distributions. Experimental results with 150 Beatles songs that has keys and no modulation showed that the recognition rate was 73.7% on average. © 2012 Springer-Verlag.

Affordance theory suggests that humans recognize the environment based on invariants. Invariants are features that describe the environment offering behavioral information to humans. Two types of invariants exist, structural invariants and transformational invariants. In our previous paper, we developed a method that self- organizes transformational invariants, or motion features, from camera images based on robot's experiences. The model used a bi-directional technique combining a recurrent neural network for dynamics learning, namely Recurrent Neural Network with Parametric Bias (RNNPB), and a hierarchical neural network for feature extraction. The bi-directional training method developed in the previous work was effective in clustering the motion of objects, but the analysis did not give good segregation results of the self-organized features (transformational invariants) among different motion types. In this paper, we present a refined model which integrates dynamics learning and feature extraction in a single model. The refined model is comprised of Multiple Timescales Recurrent Neural Network (MTRNN), which possesses better learning capability than RNNPB. Self-organization result of four types of motions have proved the model's capability to create clusters of object motions. The analysis showed that the model extracted feature sequences with different characteristics for four object motion types. © 2012 IEEE.

We propose neural models for segmenting the area of a body from visual scene based on predictability. Neuroscience has shown that a prediction model in brain, which predicts sensory-feedback from motor command, can divide the sensory-feedback into the self-motion derived feedback and other derived feedback. The prediction model is important for prediction control of the body. Previous studies in robotics of the prediction model assumed that a robot can recognize the position of its body (e.g. its hand) and that the view contains only that body part. In our models, motor commands and visual feedback (pixel image that includes not only a hand but also object and background) are input into a neural network model and then the body area is segmented and prediction model of body is acquired. Our model contains two parts: 1) An object detection model obtains a conversion system between object positions and the pixel image. 2) A movement prediction model predicts hand-object positions from motor commands and identifies the body. We confirmed that our models can segment the body/object area based on their pixel textures and discriminate between them by using prediction error. © 2012 IEEE.

This paper presents a state space model for a multiperson ensemble and an estimation method of the onset timings, tempos, and leaders. In a multiperson ensemble, determining one explicit leader is difficult because (1) participants' rhythms are mutually influenced and (2) they compete with each other. Most ensemble studies however assumed that one leader exists at a time and the others just follow the leader. To deal with the multiple and time-varying leaders, we define leaderness indicating the power to influence the others as the product of the tempo stability and the distance from the ensemble tempo. This definition means that a leader should have a strong desire to change the current tempo. Using the leaderness, we present a state space model of a multiperson ensemble and an unscented Kalman filter based estimation method. The model consists of the leaderness update, the ensemble tempo update, the individual tempo update, and the onset timing adaptation, each of which has a relationship to psychological results of an ensemble. We evaluate our method using simulation and human behavior. The simulation results show that our model is stable for various initial tempos and the number of participants. For the human behavior, pairs and triads of participants are asked to tap keys in synchronization with the others. The results show that the leaderness successfully indicate the dynamics of the leaders, and the onset errors are 181msec and 241msec for pairs and triads on average, respectively, which are comparable to those of humans (153msec and 227msec for pairs and triads, respectively.) © 2012 IEEE.

Our goal is to create an ensemble between human guitarists and music robots, e.g., singing and playing instruments robots. Such robots need to detect the tempo and beat time of the music. Score following and beat tracking, which requires and does not requires a score, are commonly used for this purpose. Score following is an incremental audio-to-score alignment. Although most score following methods assume that players have a precise score, most scores for guitarists have only melody and chord sequences without any beat patterns. An audio-visual beat tracking for guitarists is reported that improves the accuracy of beat detection. However, the result of this method is still insufficient because it uses only onset information, not pitch information, and because the hand tracking shows low accuracy. In this paper, we report a multimodal score following for a guitar performance, an extension of an audio-visual beat tracking method. The main difference is to use chord sequences to improve tracking of audio signals and depth information to improve tracking of guitar playing. Chord sequences are used for the calculation of chord correlation between the input and a score. Depth information is used in the guitar plane masking by three dimensional Hough transform, for the stable detection of a hand. Finally, the system extracts score positions and tempos by a particle-filter based integration of audio and visual features, The resulting score following system improves the tempo and the score position of a performance by 0.2 [sec] compared to an existing system. © 2012 IEEE.

聞く，見るは，共演者が仲間の演奏者とテンポを合わせて演奏するために重要なスキルである．画像キュー（cue，合図）を検知し，他の演奏者に耳を傾けることによって，演奏者はいっせいに演奏を始め，テンポの緩急の合図に合わせ，さらに，いっせいに演奏を終えることができる．本稿では，人間のフルート奏者がアンサンブルリーダを担い，ロボットは伴奏者として人間の演奏に追従する問題を扱う．まず，フルート奏者の3種類のジェスチャを提案し，画像キューによる認識，音響ビートと画像キューとの統合によるテンポ推定について述べ，テルミン演奏共演ロボットのジェスチャ認識について報告する．初期実験で3タイプの画像キューが83%以上で検出できること，また，画像キューと音響ビート検出とを組み合わせることにより，テンポ検出が0.5秒以内に行えればビート追跡が安定することが分かった．この結果，フルート奏者の指示に合わせて共演者音楽ロボットがテルミンを演奏し，歌を歌うことが可能となった．Listening and watching are important skills for co-players to play in time with fellow musicians. By detecting visual cues and listening to other players, musicians can start together, stop together, and follow a leader's visual cues of changes in tempo. In this paper, we formalize three visual cues for the case of flutists, and describe how our thereminist robot co-player system detects them. Initial experiments show over 83% detection rates for our 3 types of visual cues. Additionally, by coupling visual cues and acoustic beat detection, the robot can extract a tempo in half a second. The resulting robot co-player can play theremin and sing a song with the lead of a human flutist.

本稿では、楽器パート毎に、楽譜と音響信号のアライメントを算出する手法を提案する。本手法では、各楽器パートに共通の、自己回帰過程に従うテンポモデルを持たせる。各楽器パートの時系列は隠れセミマルコフモデルに従い、状態継続長の事前分布としてテンポモデルを持つ。また、音響信号の出力は潜在的調波配分法に従う。パート間の揺らぎを持たせない場合の、アライメントの性能を評価し、アライメント手法としての有用性が確認された。また、演奏における発音タイミングの揺らぎがモデル化できることが示唆された。This paper presents a method to align an audio signal and individual music instrument parts comprising a music score. Such method allows a machine to analyze temporal interaction of music performers. Proposed method is based on fitting multiple Hidden Semi-Markov Models (HSMM) to the observed audio signal, each HSMM of which emits Latent Harmonic Allocation parameters. Each HSMM corresponds to a music instrument part, and the state duration probability is conditioned on an auto-regressive tempo model. Evaluation suggests usefulness as score alignment method, and hints at the usefulness as multiple part alignment method.

Our goal is to crate a robot which can communicate with people for a long time. To make this possible, we are developing a customizable communication robot "WEAR (Waseda Extendable ARchitecture)". We also provided modules, and experimental subjects can customize the robot by inserting them into the module-port. At the same time, the robot can recognize the customize, and it also can express its autonomy by rejecting the customization by ejecting the module. In this paper, we proposed the communication model to grow self-efficacy. And as the result of the experiments, we found that the proposed communication model is preferred by user than all-accept communication model.

We show that a Multiple Timescale Recurrent Neural Network (MTRNN) can acquire the capabilities to recognize, generate, and correct sentences by self-organizing in a way that mirrors the hierarchical structure of sentences: characters grouped into words, and words into sentences. The model can control which sentence to generate depending on its initial states (generation phase) and the initial states can be calculated from the target sentence (recognition phase). In an experiment, we trained our model over a set of unannotated sentences from an artificial language, represented as sequences of characters. Once trained, the model could recognize and generate grammatical sentences, even if they were not learned. Moreover, we found that our model could correct a few substitution errors in a sentence, and the correction performance was improved by adding the errors to the training sentences in each training iteration with a certain probability. An analysis of the neural activations in our model revealed that the MTRNN had self-organized, reflecting the hierarchical linguistic structure by taking advantage of the differences in timescale among its neurons: in particular, neurons that change the fastest represented "characters", those that change more slowly, "words", and those that change the slowest, "sentences". © 2011 Elsevier Ltd.

合奏において、ビートトラッキングは動作タイミングの取得の基礎となる技術である。ギターとの合奏において、ビートトラッキングは演奏テンポの揺らぎや裏拍ビートを含む多様なリズムへの頑健性、つまり(1)テンポと(2)音符長の両方の変動に対する追従性が要求される。しかし従来の手法では両立できなかった。本研究では視聴覚情報統合により、両者の変動追従性向上を実現する。(1)の問題にはSTPMという聴覚情報を用いた手法を適用する。(2)の問題はギター演奏動作の周期性を利用し手の位置情報を取得、それとSTPMで得られる信頼度関数とに粒子フィルタを適用することで解決する。

我々はマイク数より音源数が多い状況(劣決定状況)に対処するため, 伝達関数が既知の際にL1ノルム最小化を用いて音源分離を行なう手法を改良し, その有効性を確認してきた. しかし我々の手法は高速化のために線形計画(LP)による近似計算を利用しており, 理論的裏付けのある二次錐計画(SOCP)との誤差が提案手法の性能を低下させている可能性があった. 本稿ではLPによる近似解とSOCPによる厳密解を比較し, LPによる近似解が十分に実用可能な精度を保っていることを確認した.

Is this person playing a violin or a flute? Classification of musical instrument performances is usually carried out using audio features such as spectral coefficients. We propose augmenting the typical audio feature set with visual features. We show that a combination of audio features and video perform better than audio alone, and verify this multimodal recognition approach on a real-time robot platform.

パーティクルフィルタによる楽譜追従は，音響信号と楽譜との距離に基づくパーティクル重みの計算によって追従性能が大きく左右される．従来のベクトル内積計算やシグモイド関数を用いた重み計算手法では，音響信号の非調波成分や楽器の音色のバリエーションにより，楽譜位置推定が正しい場合，誤った場合でそれぞれの重みに大きな差が生じず，最終的に推定された楽譜位置に誤差が含まれるという問題点があった．本稿では，過去に計算された距離の累積頻度から重みを動的に計算し，正しい楽譜位置ではより高い重みを計算する．評価実験では，累積頻度を用いた重み計算法が，従来の重み計算法よりも楽譜追従精度で改善することが確認された．

本稿では楽器音響信号に加えられたディレイエフェクトの強度及び時刻の推定手法について述べる。ディレイ信号は残響の一種と見なせるが、反響音の生起が連続的でなく高々数回しかない点が通常の残響と異なるため、従来の残響推定アルゴリズムでは推定精度が不十分である。我々は、ディレイ音を記述する伝達関数の係数がスパースであるという制約条件の元で残響推定を行うことにより、ディレイエフェクトの強度及び時刻を高精度に推定する。

本稿では調波パラメトリックNonnegative Matrix Factorization (HPNMF) と呼ぶ新しい振幅スペクトログラムモデリング手法を用いた音源分離と演奏合成法について述べる．HPNMFでは，振幅スペクトログラムを直接因子分解するのではなく，各時刻のスペクトルを調波Gaussian Mixtureによりモデル化した上でその各倍音強度パラメータを楽曲全体で因子分解する．これにより基本周波数パラメータをNMFの枠組みの外側で適応でき，通常のNMFが苦手とするビブラート信号などを効率的に推定できる．HPNMFを用いて多重奏からの特定楽器パート音源分離と演奏音響信号再合成が高精度に実現されることを示す．

Keyboard playing is a widely used method to represent musical idea,which is played in either harmonic or textural styles. The goal ofthis paper is to classify such style of the user's keyboard playingfrom its audio signal. Because the acoustic features for suchclassification is poorly studied, we defined acoustic features whichrepresent harmonic and textural playing style, and classified actualpractical keyboard playings.

This paper describes a speaker localization system using two microphonesembedded in the head of the SIG-2 humanoid robot. Nowadays, studies based onmicrophone arrays utilize many microphones to obtain high detectionperformance in acoustical engineering. However, using multiple microphoneswould raise costs and hinder maintenance. To achieve good results in speakerlocalization only with two low-cost microphones, we employ the statisticalmodel-based voice activity detection (VAD) and a sound-source-directionestimator based on the generalized cross-correlation with phase transform(GCC-PHAT). The experimental results show that SIG-2 can localize a callerreliably when the speaker calls robot's name in the azimuth.

歌声，話声，歌声と話声の中間的な音声，歌舞伎や能の音声のそれぞれの歌声らしさ，話声らしさを評価する．人間の歌声らしさ・話声らしさに関する知覚は連続的に変化すると考えられるため，中間的に知覚される音声が存在する．従来の研究では歌声か，話声かのみを考えており，中間的な音声や歌声・話声以外を考慮していない．計算機で中間的な音声の評価ができれば人間の歌声らしさ・話声らしさの知覚機構解明に貢献できると考えられる．本報告ではF0・音韻長・パワーを制御することで中間的な音声を作る．合成した音声に対し聴取実験を行い，どのような制御が歌声らしさ・話声らしさ知覚に影響を与えるのか聴取実験で評価する．

本稿では，再帰結合型神経回路モデルへのスパース結合導入による性能向上を示す．近年，多様な時系列パターンを学習可能な，スパース結合型神経回路が着目を集めている．しかし通常これらのモデルは内部の結合重みが固定されており，学習能力には限界がある．我々は，異なる時定数のニューロン群からなる再帰結合型神経回路モデル，MTRNNの一部結合をスパース化し，全結合を学習可能としたモデルの性能評価を行った．スパース化率の異なるMTRNNに，アルファベット列からなる文章を学習させ，未知文及びノイズ文の認識・生成能力の評価を行った．実験の結果，スパース結合とすることで，全結合の場合よりも性能を向上できることが確認された．

This paper presents a new method based on independent component analysis (ICA) for enhancing a target source and suppressing other interfering sound sources, supposed that the latter are known. The method can provides in a reverberant environment a barge-in-able robot audition system; that is, the user can talk to the robot at any time even when the robot speaks. Our method separates and dereverberates the user&#039;s speech and the robot&#039;s one by using Multiple Input ICA. The critical issue for real-time processing is to reduce the computational complexity of Multiple Input ICA to the linear order of the reverberation time, which has not been proposed so far. We attain it by exploit the property of the independence relationship between late observed signals and late speech signals. Experimental results show that 1) the computational complexity of our method is less than the na&amp;iuml;ve Multiple Input ICA method, and that 2) our method improves word correctness of automatic speech recognition under barge-in and reverberant situations; by at most 40 points for reverberation time of 240[ms] and 30 points for 670[ms].

This paper proposes a continuous vowel imitation system that explains the process of phoneme acquisition by infants from the dynamical systems perspective. Almost existing models concerning this process dealt with discrete phoneme sequences. Human infants, however, have no knowledge of phoneme innately. They perceive speech sounds as continuous acoustic signals. The imitation target of this study is continuous acoustic signals including unknown numbers and kinds of phonemes. The key ideas of the model are (1) the use of a physical vocal tract model called the Maeda model for embodying the motor theory of speech perception, (2) the use of a dynamical system called the Recurrent Neural Network with Parametric Bias (RNNPB) trained with both dynamics of the acoustic signals and articulatory movements of the Maeda model, and (3) the segmenting method of a temporal sequence using the prediction error of the RNNPB model. The experiments of our model demonstrated following results: (a) the self-organization of the vowel structure into attractors of RNNPB model, (b) the improvement of vowel imitation using movement of the Maeda model, and (c) the generation of clear vowels based on the bubbling process trained with a few random utterances. These results suggest that our model reflects the process of phoneme acquisition.

Our goal is to develop a coplayer music robot capable of presenting a musical expression together with humans. Although many instrument-performing robots exist, they may have difficulty playing with human performers due to the lack of the synchronization function. The robot has to follow differences in humans' performance such as temporal fluctuations to play with human performers. We classify synchronization and musical expression into two levels: (1) melody level and (2) rhythm level to cope with erroneous synchronizations. The idea is as follows: When the synchronization with the melody is reliable, respond to the pitch the robot hears, when the synchronization is uncertain, try to follow the rhythm of the music. Our method estimates the score position for the melody level and the tempo for the rhythm level. The reliability of the score position estimation is extracted from the probability distribution of the score position. The experimental results demonstrate that our method outperforms the existing score following system in 16 songs out of 20 polyphonic songs. The error in the prediction of the score position is reduced by 69 on average. The results also revealed that the switching mechanism alleviates the error in the estimation of the score position. Copyright © 2011 Takuma Otsuka, et al.

Navigation robots must single out partners requiring navigation and move in the cluttered environment where people walk around. Developing such robots requires two different people detections: detecting partners and detecting all moving people around the robots. For detecting partners, we design divided spaces based on the spatial relationships and sensing ranges. Mapping the friendliness of each divided space based on the stimulus from the multiple sensors to detect people calling robots positively, robots detect partners on the highest friendliness space. For detecting moving people, we regard objects' floor boundary points in an omnidirectional image as obstacles. We classify obstacles as moving people by comparing movement of each point with robot movement using odometry data, dynamically changing thresholds to detect. Our robot detected 95.0% of partners while it stands by and interacts with people and detected 85.0% of moving people while robot moves, which was four times higher than previous methods did.

In this paper, we propose a model for recognizing written text through prediction of a handwriting sequence. The approach is based on findings in the brain sciences field. When recognizing written text, humans are said to unintentionally trace its handwriting sequence in their brains. Likewise, we aim to create a model that predicts a handwriting sequence from a static image of written text. The predicted handwriting sequence would be used to recognize the text. As the first step towards the goal, we created a model using neural networks, and evaluated the learning and recognition capability of the model using single Japanese characters. First, the handwriting image sequences for training are self-organized into image features using a self-organizing map. The self-organized image features are used to train the neuro-dynamics learning model. For recognition, we used both trained and untrained image sequences to evaluate the capability of the model to adapt to unknown data. The results of two experiments using 10 Japanese characters show the effectivity of the model. (C) Koninklijke Brill NV, Leiden and The Robotics Society of Japan, 2011

This paper presents a Bayesian method for temporally aligning a music score and an audio rendition. A critical problem in audio-to-score alignment is in dealing with the wide variety of timbre and volume of the audio rendition. In contrast with existing works that achieve this through ad-hoc feature design or careful training of tone models, we propose a Bayesian audio-to-score alignment method by modeling music performance as a Bayesian Hidden Markov Model, each state of which emits a Bayesian signal model based on Latent Harmonic Allocation. After attenuating reverberation, variational Bayes method is used to iteratively adapt the alignment, instrument tone model and the volume balance at each position of the score. The method is evaluated using sixty works of classical music of a variety of instrumentation ranging from solo piano to full orchestra. We verify that our method improves the alignment accuracy compared to dynamic time warping based on chroma vector for orchestral music, or our method employed in a maximum likelihood setting. © 2011 IEEE.

This paper presents a method of both separating audio mixtures into sound sources and identifying the musical instruments of the sources. A statistical tone model of the power spectrogram, called an integrated model, is defined and source separation and instrument identification are carried out on the basis of Bayesian inference. Since, the parameter distributions of the integrated model depend on each instrument, the instrument name is identified by selecting the one that has the maximum relative instrument weight. Experimental results showed correct instrument identification enables precise source separation even when many overtones overlap. © 2011 IEEE.

Our goal is to develop a system that is able to learn and classify environmental sounds for robots working in the real world. In the real world, two main restrictions pertain in learning. First, the system has to learn using only a small amount of data in a limited time because of hardware restrictions. Second, it has to adapt to unknown data since it is virtually impossible to collect samples of all environmental sounds. We used a neuro-dynamical model to build a prediction and classification system which can self-organize sound classes into parameters by learning samples. The proposed system searches space of parameters for classifying. In the experiment, we evaluated the accuracy of classification for known and unknown sound classes. © 2011 Springer-Verlag.

In real-world situations, people often hear more than two simultaneous sounds. For robots, when the number of sound sources exceeds that of sensors, the situation is called under-determined, and robots with two ears need to deal with this situation. Some studies on under-determined sound source separation use L1-norm minimization methods, but the performance of automatic speech recognition with separated speech signals is poor due to its spectral distortion. In this paper, a two-stage separation method to improve separation quality with low computational cost is presented. The first stage uses a L1-norm minimization method in order to extract the harmonic structures. The second stage exploits reliable harmonic structures to maintain acoustic features. Experiments that simulate three utterances recorded by two microphones in an anechoic chamber show that our method improves speech recognition correctness by about three points and is fast enough for real-time separation. © 2011 Springer-Verlag.

Our goal is to achieve a robot audition system that is capable of recognizing multiple environmental sounds and making use of them in human-robot interaction. The main problems in environmental sound recognition in robot audition are: (1) recognition under a large amount of background noise including the noise from the robot itself, and (2) the necessity of robust feature extraction against spectrum distortion due to separation of multiple sound sources. This paper presents the environmental recognition of two sound sources fired simultaneously using matching pursuit (MP) with the Gabor wavelet, which extracts salient audio features from a signal. The two environmental sounds come from different directions, and they are localized by multiple signal classification and, using their geometric information, separated by geometric source separation with the aid of measured head-related transfer functions. The experimental results show the noise-robustness of MP although the performance depends on the properties of the sound sources. © 2011 Springer-Verlag.

This paper presents an efficient algorithm to solve Lp-norm minimization problem for under-determined speech separation; that is, for the case that there are more sound sources than microphones. We employ an auxiliary function method in order to derive update rules under the assumption that the amplitude of each sound source follows generalized Gaussian distribution. Experiments reveal that our method solves the L1-norm minimization problem ten times faster than a general solver, and also solves Lp-norm minimization problem efficiently, especially when the parameter p is small; when p is not more than 0.7, it runs in real-time without loss of separation quality. Copyright © 2011 ISCA.

This paper presents a Bayesian extension of MUSIC-based sound source localization (SSL) and tracking method. SSL is important for distant speech enhancement and simultaneous speech separation for improving speech recognition, as well as for auditory scene analysis by mobile robots. One of the draw- backs of existing SSL methods is the necessity of careful param- eter tunings, e.g., the sound source detection threshold depend- ing on the reverberation time and the number of sources. Our contribution consists of (1) automatic parameter estimation in the variational Bayesian framework and (2) tracking of sound sources with reliability. Experimental results demonstrate our method robustly tracks multiple sound sources in a reverberant environment with RT20 = 840 (ms). Copyright © 2011 ISCA.

This paper presents an audio-visual beat-tracking method for ensemble robots with a human guitarist. Beat-tracking, or estimation of tempo and beat times of music, is critical to the high quality of musical ensemble performance. Since a human plays the guitar in out-beat in back beat and syncopation, the main problems of beat-tracking of a human's guitar playing are twofold: tempo changes and varying note lengths. Most conventional methods have not addressed human's guitar playing. Therefore, they lack the adaptation of either of the problems. To solve the problems simultaneously, our method uses not only audio but visual features. We extract audio features with Spectro-Temporal Pattern Matching (STPM) and visual features with optical flow, mean shift and Hough transform. Our beat-tracking estimates tempo and beat time using a particle filter; both acoustic feature of guitar sounds and visual features of arm motions are represented as particles. The particle is determined based on prior distribution of audio and visual features, respectively Experimental results confirm that our integrated audio-visual approach is robust against tempo changes and varying note lengths. In addition, they also show that estimation convergence rate depends only a little on the number of particles. The real-time factor is 0.88 when the number of particles is 200, and this shows out method works in real-time. © 2011 IEEE.

This paper presents an improved speaker localization method based on the generalized cross-correlation (GCC) method weighted by the phase transform (PHAT) for binaural robot audition. The problem with the conventional direction-of-arrival (DOA) estimation based on the GCC-PHAT method is a multipath interference whereby a sound wave travels to microphones via the front-head path and the back-head path in binaural robot audition. This paper describes a new time delay factor for the GCC-PHAT method to compensate multipath interference on the assumption of spherical robot head. In addition, the restriction of the time difference of arrival (TDOA) estimation by the sampling frequency is also solved by applying the maximum likelihood (ML) estimation in frequency domain. Experiments conducted in the SIG-2 humanoid robot show that the proposed method reduces localization errors by 17.8 degrees on average and by over 35 degrees in side directions comparing to the conventional DOA estimation. © 2011 IEEE.

Our goal is to develop a system to learn and classify environmental sounds for robots working in the real world. In the real world, two main restrictions pertain in learning. (i) Robots have to learn using only a small amount of data in a limited time because of hardware restrictions. (ii) The system has to adapt to unknown data since it is virtually impossible to collect samples of all environmental sounds. We used a neuro-dynamical model to build a prediction and classification system. This neuro-dynamical model can self-organize sound classes into parameters by learning samples. The sound classification space, constructed by these parameters, is structured for the sound generation dynamics and obtains clusters not only for known classes, but also unknown classes. The proposed system searches on the basis of the sound classification space for classifying. In the experiment, we evaluated the accuracy of classification for both known and unknown sound classes. © 2011 Koninklijke Brill NV, Leiden.

Humans are said to unintentionally trace handwriting sequences in their brains based on handwriting experiences when recognizing written text. In this paper, we propose a model for predicting handwriting sequence for written text recognition based on handwriting experiences. The model is first trained using image sequences acquired while writing text. The image features of sequences are self-organized from the images using Self-Organizing Map. The feature sequences are used to train a neuro-dynamics learning model. For recognition, the text image is input into the model for predicting the handwriting sequence and recognition of the text. We conducted two experiments using ten Japanese characters. The results of the experiments show the effectivity of the model. © 2011 IEEE.

Music information retrieval, especially the audio-to-score alignment problem, often involves a matching problem between the audio and symbolic representations. We must cope with uncertainty in the audio signal generated from the score in a symbolic representation such as the variation in the timbre or temporal fluctuations. Existing audio-to-score alignment methods are sometimes vulnerable to the uncertainty in which multiple notes are simultaneously played with a variety of timbres because these methods rely on static observation models. For example, a chroma vector or a fixed harmonic structure template is used under the assumption that musical notes in a chord are all in the same volume and timbre. This paper presents a particle filterbased audio-to-score alignment method with a flexible observation model based on latent harmonic allocation. Our method adapts to the harmonic structure for the audio-toscore matching based on the observation of the audio signal through Bayesian inference. Experimental results with 20 polyphonic songs reveal that our method is effective when more number of instruments are involved in the ensemble. © 2011 International Society for Music Information Retrieval.

In this paper we present a new method for producing affective motion for humanoid robots. The NAO robot, like other humanoids, does not possess facial features to convey emotion. Instead, our proposed system generates pose-independent robot movement using a description of emotion through speed, intensity, regularity and extent (DESIRE). We show how the DESIRE framework can link the emotional content of voice and gesture, without the need for an emotion recognition system. Our results show that DESIRE movement can be used to effectively convey at least four emotions with user agreement 60-75%, and that voices converted to motion through SIRE maintained the same emotion significantly higher than chance, even across cultures (German to Japanese). Additionally, portrayals recognized as happiness were rated significantly easier to understand with motion over voice alone. © 2011 IEEE.

In this paper, we present a new method that represents an overall musical time-varying impression of a song by a pair of mood trajectories estimated from lyrics and audio signals. The mood trajectory of the lyrics is obtained by using the probabilistic latent semantic analysis (PLSA) to estimate topics (representing impressions) from words in the lyrics. The mood trajectory of the audio signals is estimated from acoustic features by using the multiple linear regression analysis. In our experiments, the mood trajectories of 100 songs in Last.fm's Best of 2010 were estimated. The detailed analysis of the 100 songs confirms that acoustic features provide more accurate mood trajectory and the 21% resulting mood trajectories are matched to realistic musical mood available at Last.fm. © 2011 ACM.

The objective of our study is to find out how a sparse structure affects the performance of a recurrent neural network (RNN). Only a few existing studies have dealt with the sparse structure of RNN with learning like Back Propagation Through Time (BPTT). In this paper, we propose a RNN with sparse connection and BPTT called Multiple time scale RNN (MTRNN). Then, we investigated how sparse connection affects generalization performance and noise robustness. In the experiments using data composed of alphabetic sequences, the MTRNN showed the best generalization performance when the connection rate was 40%. We also measured sparseness of neural activity and found out that sparseness of neural activity corresponds to generalization performance. These results means that sparse connection improved learning performance and sparseness of neural activity would be used as metrics of generalization performance. © 2011 Springer-Verlag.

We propose a novel navigation system for adaptively exploring an obstacle space using diverse ways of touching an object. Conventional navigation models are typically based on the avoidance of obstacles, i.e., avoiding collision. However, actual disordered space may be full of various kinds of obstacles. To reach a destination in such a space, a robot requires an active approach for avoiding a deadlock with obstacles or changing the obstacle configuration to find an open space using diverse ways of touching an object. We solved this problem by generating locally diverse moving patterns by using an action model with rhythmical oscillation in addition to a localization model using a particle filter. The proposed model was demonstrated to be effective through an experiment where a robot navigated to a destination behind partially movable obstacles using rhythmical active touch. © 2011 IEEE.

The goal of our work is to create natural verbal interaction between humans and speech dialogue agents. In this paper, we focus on generations of back-channel for speech dialogue agents the same way humans do. To create such a system, the system needs to predict the appropriate timing of back-channel on the basis of the human's speech. For the prediction model, we use a neuro-dynamical system called a multiple timescale recurrent neural network (MTRNN). The model is trained using an actual corpus of a poster session of the IMADE project using the presenter's prosodic and visual information as features. Using the model, we conducted back-channel timing prediction experiments. The results showed that our system could predict back-channel timing about 0.5 seconds before generation of back-channel response. Comparing the results with the actual back-channel timing in the corpus, the system showed 37.1% of recall, 31.7% of precision, and 34.2% of F-measure. These results show the model to effectively predict and generate back-channel responses. © 2011 IEEE.

The goal of our work is to acquire an internal model through a robot's experience. The internal model has the ability for mutual conversion between motor commands and movement of the body (e.g. hand) in view. Unlike other works, which assume the robot's body to be extracted in its view, we assume that external moving objects are also included in its view. We introduce predictability as a measure to segregate such objects from the robot's body: the robot's body is predictable while moving objects are not. Prediction is conducted using a neuro-dynamical system called the multiple timescales recurrent neural network (MTRNN). The prediction results of the robot's body are compared with the actual motion to distinguish the robot's body from other objects. For evaluation, we conducted an experiment with the robot moving its hand while moving objects were in view. The results of the experiment showed that the prediction of the robot's hand is 3.86 times as accurate as that of others on average. These results show the effectiveness of using predictability as a measure to acquire an internal model in an environment that includes both a robot's body and other moving objects in view. © 2011 IEEE.

We address the issue of out-of-grammar (OOG) utterances in spoken dialogue systems by generating help messages. Help message generation for OOG utterances is a challenge because language understanding based on automatic speech recognition (ASR) of OOG utterances is usually erroneous; important words are often misrecognized or missing from such utterances. Our grammar verification method uses a weighted finite-state transducer, to accurately identify the grammar rule that the user intended to use for the utterance, even if important words are missing from the ASR results. We then use a ranking algorithm, RankBoost, to rank help message candidates in order of likely usefulness. Its features include the grammar verification results and the utterance history representing the user's experience. Copyright © 2010.

Our goal is to create a robot which can communicate with people for a long time. To make this possible, we are developing a customizable communication robot "WEAR (Waseda Extendable ARchitecture)". We also provided modules, and experimental subjects can customize the robot by inserting them into the module-port. At the same time, the robot can recognize the customize, and it also can express its autonomy by rejecting the customization by ejecting the module. In this paper, we propose to introduce a requesting behavior (making sound and moving the module-port) into the robot as well as the rejecting behavior. As the result of experiments, we found that the impression of the sound differs among subjects in the view point of understanding the robot's request.

本論文では，テルミンを演奏するロボットのためのテルミンの特性モデルと演奏動作生成手法について報告する．テルミンとは，演奏者の手の位置を動かして演奏する電子楽器である．楽器との物理的接触なしで連続的に音高と音量を操作できるので，ハードウェア構成が異なるロボットにも適用可能であるという点で，移植性が高い．テルミン演奏ロボットの主たる課題は，(1)動作生成の物理的な基準点がないので，演奏法学習に要する学習サンプル数が多くなること，および(2)演奏特性が静電的環境によって変化するので，適応的な演奏動作生成が必要であることの2点である．これらの課題に対して，我々は環境の影響をパラメータで表現した音高・音量特性モデルを構築し，少数の測定で音域内の任意の音を演奏できる制御手法を開発した．実験の結果，約12点の測定で音高が任意に制御できること，環境が変化しても所望の音高や音量で演奏できることを3種類のロボットで確認した．We present a theremin player robot towards an ensemble between humans and robots. A theremin, whose pitch and volume change continuously, can be played without any physical contacts. We thus expect that a robot system has high portability because it requires only few physical constraints. The problems for theremin playing are: (1) we have no physical reference points and (2) an environment affects sound characteristics seriously. To solve them, we develop a model-based feedforward arm control method based on our novel models of theremin's pitch and volume characteristics, which method realizes play an arbitrary sound using a few measurements. Experimental results show that our method works under four environments and three different robots.

A system for mapping between different sensory modalities was developed for a robot system to enable it to generate motions expressing auditory signals and sounds generated by object movement. A recurrent neural network model with parametric bias, which has good generalization ability, is used as a learning model. Since the correspondences between auditory signals and visual signals are too numerous to memorize, the ability to generalize is indispensable. This system was implemented in the "Keepon" robot, and the robot was shown horizontal reciprocating or rotating motions with the sound of friction and falling or overturning motion with the sound of collision by manipulating a box object. Keepon behaved appropriately not only from learned events but also from unknown events and generated various sounds in accordance with observed motions. © 2009 Elsevier B.V. All rights reserved.

本稿は，バージイン許容列挙型音声対話におけるユーザ発話の分析と，分析結果を応用した指示対象同定手法の拡張について報告する．バージイン許容音声対話では，個々のユーザやシステムの発話内容によってユーザの発話タイミングや発話表現が異なる．そこでこれらを事前確率として反映させ，発話意図解釈の性能向上を図る．我々はまず，ニュース読み上げとクイズの2つの列挙型対話システムで収集したユーザ発話 1584 発話を分析し，ユーザの参照表現発話率が個々のユーザやシステムの列挙項目長に依存することを明らかにした．さらに，これらの特性を指示対象同定の枠組みに組み込み，タイミングと音声認識結果の解釈の事前確率として反映させる．この事前確率の推定には，ロジスティック回帰を用いる．事前確率として一定値を用いた場合に比べて，指示対象同定精度が最大 6.2 ポイント向上することを実験により確認した．This paper reports the extension of identification method based on analyses of user utterance in barge-in-able spoken dialogue system which reads out items. Generally, user&#039;s behaviors such as barge-in timing and utterance expressions vary in accordance with the user&#039;s preference and the content of system utterances. To interpret users&#039; intention robustly, first, we analyze 1584 utterances collected by our systems with quiz and news-listing tasks and reveal that the ratio of using referential expressions depends on individual users and average lengths of listed items. Second, we incorporate this tendency as a prior probability into our probabilistic framework for identifying user&#039;s intended item. This prior probability is calculated by logistic regression. Experimental results show that our method improves the identification accuracy by as many as 6.2 points in the best case over the non-informative prior.

本論文では,音声対話システムにおいて,複数の言語モデルと複数の言語理解モデルを用いることで,高精度な音声理解を行う手法について述べる.ユーザの発話によって適した言語モデル・言語理解モデルの組合せは異なることから,単一の音声理解方式で様々な発話に対して高精度な音声理解を実現することは難しい.そこで本論文では,まず,複数の言語モデルと言語理解モデルを用いて複数の理解結果を得ることで,理解結果の候補を得る.次に,得られた複数の理解結果に対して,ロジスティック回帰に基づき発話単位の信頼度を付与し,その信頼度が最も高い理解結果を選択する.本論文では,言語モデルとして文法モデルとN-gramモデルの2種類,言語理解モデルとしてFinite-State Transducer(FST)とWeighted FST(WFST),Keyphrase-Extractorの3種類を用いた.評価実験では,言語モデル・言語理解モデルのいずれかを複数用いた場合と比較して,コンセプト理解精度の向上が得られた.また,従来のROVER法による理解結果の統合と比較し,本手法の有効性を示した.

本稿では，少量の学習データでも高精度な音声理解を実現する手法について述べる．学習データが少ない場合，単一の音声理解方式による精度は低い傾向にある．そこで本手法では，まず，複数の言語モデルと言語理解モデルを用いて複数の理解結果を得ることで，対処可能な発話を増やす．次に，得られた複数の理解結果に対して，ロジスティック回帰に基づき発話単位の信頼度を付与し，その信頼度が最も高い理解結果を選択する．ロジスティック回帰には，学習データ増加時の回帰係数の変化量に着目することで，必要最低限の学習データを割り当てる．評価実験では，学習データが少ない場合でも，単一の音声理解方式と比較して，本手法が高い音声理解精度を得られることを示す．We aim to improve a speech understanding module with a small amount of training data. High performance is not obtained by single speech understanding methods especially when the amount of available training data is small. We utilize multiple language models (LMs) and language understanding models (LUMs) to cover various user utterances. Then, the most appropriate speech understanding result is selected from several candidates on the basis of confidence measures calculated by logistic regressions. We determine necessary amount of training data for the regressions by focusing on changes in their coefficients when the training data increases. We evaluate our method for various amounts of training data and confirm that our method outperforms every single speech understanding method even when only a small amount of training data is available.

Our goal is to create a robot which can communicate with people for a long time. We developed the customizable communication robot "WEAR" which is installed the system which can reject of user's customization based on its judgement. In this paper, we carried out an experiment to verify the user's impression when WEAR changes it's motion based on user's understanding of the judgement. As the results, we confirmed that when WEAR changes it's motion, users more interested in WEAR, compared to didn't changes motion.

This paper presents the upper-limit evaluation of robot audition based on ICA-BSS in multi-source, barge-in and highly reverberant conditions. The goal is that the robot can automatically distinguish a target speech from its own speech and other sound sources in a reverberant environment. We focus on the multi-channel semi-blind ICA (MCSB-ICA), which is one of the sound source separation methods with a microphone array, to achieve such an audition system because it can separate sound source signals including reverberations with few assumptions on environments. The evaluation of MCSB-ICA has been limited to robot's speech separation and reverberation separation. In this paper, we evaluate MCSB-ICA extensively by applying it to multi-source separation problems under common reverberant environments. Experimental results prove that MCSB-ICA outperforms conventional ICA by 30 points in automatic speech recognition performance. ©2010 IEEE.

This paper describes improvement of sound source separation for a simultaneous automatic speech recognition (ASR) system of a humanoid robot. A recognition error in the system is caused by a separation error and interferences of other sources. In separability, an original geometric source separation (GSS) is improved. Our GSS uses a measured robot's head related transfer function (HRTF) to estimate a separation matrix. As an original GSS uses a simulated HRTF calculated based on a distance between microphone and sound source, there is a large mismatch between the simulated and the measured transfer functions. The mismatch causes a severe degradation of recognition performance. Faster convergence speed of separation matrix reduces separation error. Our approach gives a nearer initial separation matrix based on a measured transfer function from an optimal separation matrix than a simulated one. As a result, we expect that our GSS improves the convergence speed. Our GSS is also able to handle an adaptive step-size parameter. These new features are added into open source robot audition software (OSS) called "HARK" which is newly updated as version 1.0.0. The HARK has been installed on a HRP-2 humanoid with an 8-element microphone array. The listening capability of HRP-2 is evaluated by recognizing a target speech signal which is separated from a simultaneous speech signal by three talkers. The word correct rate (WCR) of ASR improves by 5 points under normal acoustic environments and by 10 points under noisy environments. Experimental results show that HARK 1.0.0 improves the robustness against noises. ©2010 IEEE.

We describe a novel dialogue strategy enabling robust interaction under noisy environments where automatic speech recognition (ASR) results are not necessarily reliable. We have developed a method that exploits utterance timing together with ASR results to interpret user intention, that is, to identify one item that a user wants to indicate from system enumeration. The timing of utterances containing referential expressions is approximated by Gamma distribution, which is integrated with ASR results by expressing both of them as probabilities. In this paper, we improve the identification accuracy by extending the method. First, we enable interpretation of utterances including ordinal numbers, which appear several times in our data collected from users. Then we use proper acoustic models and parameters, improving the identification accuracy by 4.0% in total. We also show that Latent Semantic Mapping (LSM) enables more expressions to be handled in our framework. © 2010 Springer-Verlag.

This work presents an automated violin fingering estimation method that facilitates a student violinist acquire the "sound" of his/her favorite recording artist created by the artist's unique fingering. Our method realizes this by analyzing an audio recording played by the artist, and recuperating the most playable fingering that recreates the aural characteristics of the recording. Recovering the aural characteristics requires the bowed string estimation of an audio recording, and using the estimated result for optimal fingering decision. The former requires high accuracy and robustness against the use of different violins or brand of strings; and the latter needs to create a natural fingering for the violinist. We solve the first problem by detecting estimation errors using rule-based algorithms, and by adapting the estimator to the recording based on mean normalization. We solve the second problem by incorporating, in addition to generic stringed-instrument model used in existing studies, a fingering model that is based on pedagogical practices of violin playing, defined on a sequence of two or three notes. The accuracy of the bowed string estimator improved by 21 points in a realistic situation (38% → 59%) by incorporating error correction and mean normalization. Subjective evaluation of the optimal fingering decision algorithm by seven violinists on 22 musical excerpts showed that compared to the model used in existing studies, our proposed model was preferred over existing one (p=0.01), but no significant preference towards proposed method defined on sequence of two notes versus three notes was observed (p=0.05). © 2010 Springer-Verlag.

Our goal is to achieve a musical ensemble among a robot and human musicians where the robot listens to the music with its own microphones. The main issues are (1) robust beat-tracking since the robot hears its own generated sounds in addition to the accompanied music, and (2) robust synchronizing its performance with the accompanied music even if humans' musical performance fluctuates. This paper presents a music-ensemble Thereminist robot implemented on the humanoid HRP-2 with the following three functions: (1) self-generated Theremin sound suppression by semi-blind Independent Component Analysis, (2) beat tracking robust against tempo fluctuation in humans' performance, and (3) feedforward control of Theremin pitch. Experimental results with a human drummer show the capability of this robot for the adaptation to the temporal fluctuation in his performance. © 2010 Springer-Verlag.

We show that a Multiple Timescale Recurrent Neural Network (MTRNN) can acquire the capabilities of recognizing and generating sentences by self-organizing a hierarchical linguistic structure. There have been many studies aimed at finding whether a neural system such as the brain can acquire languages without innate linguistic faculties. These studies have found that some kinds of recurrent neural networks could learn grammar. However, these models could not acquire the capability of deterministically generating various sentences, which is an essential part of language functions. In addition, the existing models require a word set in advance to learn the grammar. Learning languages without previous knowledge about words requires the capability of hierarchical composition such as characters to words and words to sentences, which is the essence of the rich expressiveness of languages. In our experiment, we trained our model to learn language using only a sentence set without any previous knowledge about words or grammar. Our experimental results demonstrated that the model could acquire the capabilities of recognizing and deterministically generating grammatical sentences even if they were not learned. The analysis of neural activations in our model revealed that the MTRNN had self-organized the linguistic structure hierarchically by taking advantage of differences in the time scale among its neurons, more concretely, neurons that change the fastest represented "characters," those that change more slowly represented "words," and those that change the slowest represented "sentences." © 2010 Springer-Verlag.

Our goal is to develop an interactive music robot, i.e., a robot that presents a musical expression together with humans. A music interaction requires two important functions: synchronization with the music and musical expression, such as singing and dancing. Many instrument-performing robots are only capable of the latter function, they may have difficulty in playing live with human performers. The synchronization function is critical for the interaction. We classify synchronization and musical expression into two levels: (1) the rhythm level and (2) the melody level. Two issues in achieving two-layer synchronization and musical expression are: (1) simultaneous estimation of the rhythm structure and the current part of the music and (2) derivation of the estimation confidence to switch behavior between the rhythm level and the melody level. This paper presents a score following algorithm, incremental audio to score alignment, that conforms to the two-level synchronization design using a particle filter. Our method estimates the score position for the melody level and the tempo for the rhythm level. The reliability of the score position estimation is extracted from the probability distribution of the score position. Experiments are carried out using polyphonic jazz songs. The results confirm that our method switches levels in accordance with the difficulty of the score estimation. When the tempo of the music is less than 120 (beats per minute; bpm), the estimated score positions are accurate and reported; when the tempo is over 120 (bpm), the system tends to report only the tempo to suppress the error in the reported score position predictions. Copyright © 2010, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.

The optimal choice of speech understanding method depends on the amount of training data available in rapid prototyping. A statistical method is ultimately chosen, but it is not clear at which point in the increase in training data a statistical method become effective. Our framework combines multiple automatic speech recognition (ASR) and language understanding (LU) modules to provide a set of speech understanding results and selects the best result among them. The issue is how to allocate training data to statistical modules and the selection module in order to avoid overfitting in training and obtain better performance. This paper presents an automatic training data allocation method that is based on the change in the coefficients of the logistic regression functions used in the selection module. Experimental evaluation showed that our allocation method outperformed baseline methods that use a single ASR module and a single LU module at every point while training data increase.

In our barge-in-able spoken dialogue system, the user's behaviors such as barge-in timing and utterance expressions vary according to his/her characteristics and situations. The system adapts to the behaviors by modeling them. We analyzed 1584 utterances collected by our systems of quiz and news-listing tasks and showed that ratio of using referential expressions depends on individual users and average lengths of listed items. This tendency was incorporated as a prior probability into our method and improved the identification accuracy of the user's intended items. © 2010 ISCA.

Research on environmental sound recognition has not shown great development in comparison with that on speech and musical signals. One of the reasons is that the sound category of environmental sounds covers a broad range of acoustical natures. We classified them in order to explore suitable recognition techniques for each characteristic. We focus on impulsive sounds and their non-stationary feature within and between analytic frames. We used matching-pursuit as a framework to use wavelet analysis for extracting temporal variation of audio features inside a frame. We also investigated the validity of modeling decaying patterns of sounds using Hidden markov models. Experimental results indicate that sounds with multiple impulsive signals are recognized better by using time-frequency analyzing bases than by frequency domain analysis. Classification of sound classes with a long and clear decaying pattern improves when HMMs with multiple number of hidden states are applied. © 2010 ISCA.

This paper describes a singing-to-speaking synthesis system called "SpeakBySinging" that can synthesize a speaking voice from an input singing voice and the song lyrics. The system controls three acoustic features that determine the difference between speaking and singing voices: the fundamental frequency (F0), phoneme duration, and power (volume). By changing these features of a singing voice, the system synthesizes a speaking voice while retaining the timbre of the singing voice. The system first analyzes the singing voice to extract the F0 contour, the duration of each phoneme of the lyrics, and the power. These features are then converted to target values that are obtained by feeding the lyrics into a traditional text-to-speech (TTS) system. The system finally generates a speaking voice that preserves the timbre of the singing voice but has speech-like features. Experimental results show that SpeakBySinging can convert singing voices into speaking voices whose timbre is almost the same as the original singing voices.

The objective of our study was to develop dynamic collaboration between a human and a robot. Most conventional studies have created pre-designed rule-based collaboration systems to determine the timing and behavior of robots to participate in tasks. Our aim is to introduce the confidence of the task as a criterion for robots to determine their timing and behavior. In this paper, we report the effectiveness of applying reproduction accuracy as a measure for quantitatively evaluating confidence in an object arrangement task. Our method is comprised of three phases. First, we obtain human-robot interaction data through the Wizard of OZ method. Second, the obtained data are trained using a neuro-dynamical system, namely, the Multiple Time-scales Recurrent Neural Network (MTRNN). Finally, the prediction error in MTRNN is applied as a confidence measure to determine the robot's behavior. The robot participated in the task when its confidence was high, while it just observed when its confidence was low. Training data were acquired using an actual robot platform, Hiro. The method was evaluated using a robot simulator. The results revealed that motion trajectories could be precisely reproduced with a high degree of confidence, demonstrating the effectiveness of the method. ©2010 IEEE.

In real-world situations, a robot may often encounter "under- determined" situation, where there are more sound sources than microphones. This paper presents a speech separation method using a new constraint on the harmonic structure for a simultaneous speech-recognition system in under-determined conditions. The requirements for a speech separation method in a simultaneous speech-recognition system are (1) ability to handle a large number of talkers, and (2) reduction of distortion in acoustic features. Conventional methods use a maximum likelihood estimation in sound source separation, which fulfills requirement (1). Since it is a general approach, the performance is limited when separating speech. This paper presents a two-stage method to improve the separation. The first stage uses maximum likelihood estimation and extracts the harmonic structure, and the second stage exploits the harmonic structure as a new constraint to achieve requirement (2). We carried out an experiment that simulated three simultaneous utterances using impulse responses recorded by two microphones in an anechoic chamber. The experimental results revealed that our method could improve speech recognition correctness by about four points. ©2010 IEEE.

Predictability is an important factor for determining robot motions. This paper presents a model to generate robot motions based on reliable predictability evaluated through a dynamics learning model which self-organizes object features. The model is composed of a dynamics learning module, namely Recurrent Neural Network with Parametric Bias (RNNPB), and a hierarchical neural network as a feature extraction module. The model inputs raw object images and robot motions. Through bi-directional training of the two models, object features which describe the object motion are self-organized in the output of the hierarchical neural network, which is linked to the input of RNNPB. After training, the model searches for the robot motion with high reliable predictability of object motion. Experiments were performed with the robot's pushing motion with a variety of objects to generate sliding, falling over, bouncing, and rolling motions. For objects with single motion possibility, the robot tended to generate motions that induce the object motion. For objects with two motion possibilities, the robot evenly generated motions that induce the two object motions. ©2010 IEEE.

We describe integration of preprocessing and automatic speech recognition based on Missing-Feature-Theory (MFT) to recognize a highly interfered speech signal, such as the signal in a narrow angle between a desired and interfered speakers. As a speech signal separated from a mixture of speech signals includes the leakage from other speech signals, recognition performance of the separated speech degrades. An important problem is estimating the leakage in time-frequency components. Once the leakage is estimated, we can generate missing feature masks (MFM) automatically by using our method. A new weighted sigmoid function is introduced for our MFM generation method. An experiment shows that a word correct rate improves from 66 % to 74 % by using our MFM generation method tuned by a search base approach in the parameter space. ©2010 IEEE.

Musicians often have the following problem: they have a music score that requires 2 or more players, but they have no one with whom to practice. So far, score-playing music robots exist, but they lack adaptive abilities to synchronize with fellow players' tempo variations. In other words, if the human speeds up their play, the robot should also increase its speed. However, computer accompaniment systems allow exactly this kind of adaptive ability. We present a first step towards giving these accompaniment abilities to a music robot. We introduce a new paradigm of beat tracking using 2 types of sensory input - visual and audio - using our own visual cue recognition system and state-of-the-art acoustic onset detection techniques. Preliminary experiments suggest that by coupling these two modalities, a robot accompanist can start and stop a performance in synchrony with a flutist, and detect tempo changes within half a second. ©2010 IEEE.

This paper presents a novel synchronizing method for a human-robot ensemble using coupled oscillators. We define an ensemble as a synchronized performance produced through interactions between independent players. To attain better synchronized performance, the robot should predict the human's behavior to reduce the difference between the human's and robot's onset timings. Existing studies in such synchronization only adapts to onset intervals, thus, need a considerable time to synchronize. We use a coupled oscillator model to predict the human's behavior. Experimental results show that our method reduces the average of onset time errors; when we use a metronome, a tempo-varying metronome or a human drummer, errors are reduced by 38%, 10% or 14% on the average, respectively. These results mean that the prediction of human's behaviors is effective for the synchronized performance. ©2010 IEEE.