Recently, the deep learning technology attracts much attention in various industrial fields. In our previous research, we developed an Encoder-Decoder precisely reproducing the optimization process of conventional optimization method, that is, the level-set method which is one of the gradient methods, by means of Convolutional Neural Network (CNN) and Long Short-term Memory (LSTM). The developed method enables us to implement high speed search of solutions, which means the possibility of better and effective optimization starting with various initial shapes. This method can deal with only the initial shape as design parameter for optimization. Thus, it is necessary to re-train the Encoder-Decoder when the design conditions change, e.g., the displacement of permanent magnet. To overcome this drawback, we have developed a novel network structure to incorporate the design conditions into the training data. Finally, to confirm the validity of the proposed method, we evaluate its calculation time and computational accuracy by using a magnetic circuit design model.

In the design optimization of electric machine, we ordinarily derive the objective physical quantities, e.g., the shape of the investigated model as design variables, by using numerical method such as the finite element method with the analysis conditions. In recent years, the representation learning using Deep Learning much attracts attention because it can acquire the features of data as a distributed representation and reproduce corresponding data. In this paper, utilizing machine learning technology, we propose an application of Conditional Variational Auto-Encoder (CVAE) to reproduce the more adequate shape of magnetic materials, i.e., design variables, corresponding to the intended magnetic energy, i.e., objective function values.

The finite element analysis (FEA) of magnetic field generally requires a lot of calculation time. Especially, design optimization methods such as the level-set method with FEA result in large computational effort to find better solution. In this paper, we propose a novel method of precisely and quickly reproducing the conventional optimization steps by means of Convolutional Neural Network (CNN) and Long Short-term Memory (LSTM). The developed method enables us to implement high speed search of solution, which means the possibility of effective optimization with various initial conditions for better solution. Finally, we evaluate calculation time and computational accuracy of the proposed method by using a magnetic circuit design model.

A common approach to the electroencephalogram (EEG) source localization problem is to estimate the states of current dipoles. However, the dipole estimation problem is difficult because not only is it an inverse problem but also the number of dipoles can change over time. In this paper, we model the relationship between current dipoles and EEG observations using a state-space model where the creation and annihilation of dipoles is represented as a birth-death process. We estimate the dipoles' positions and moments with a Rao-Blackwellized particle filter and estimate whether a new dipole has been created or an existing one annihilated via the Bayesian information criterion. Experiments on both synthetic and real data show that the proposed model and estimation method can effectively estimate the number and positions of the dipoles.

Outlier detection is used to identify data points or a small number of subsets of data that are significantly different from most other data in a given dataset. It is challenging to detect outliers using an objective and quantitative approach. Methods that use the framework of statistical hypothesis testing are widely used by assuming a specific parametric distribution as a data generation model, but there is no guarantee that the distribution of data can be adequately approximated by a parametric distribution in practical problems. In this paper, a simple method is proposed to objectively detect outliers by hypothesis testing without assuming a specific distribution of outlier scores. By using an arbitrary outlier score function, hypothesis testing is used to determine whether each given sample is an outlier. The distribution of the test statistics is needed for the hypothesis test, and is estimated based on the given data using the bootstrap method. The effectiveness of the proposed outlier test was verified by applying it to outlier detection for text-based image retrieval, where it improved the quality of image searches by removing irrelevant images.

Electroencephalography (EEG) is a non-invasive brain imaging technique that describes neural electrical activation with good temporal resolution. Source localization is required for clinical and functional interpretations of EEG signals, and most commonly is achieved via the dipole model; however, the number of dipoles in the brain should be determined for a reasonably accurate interpretation. In this paper, we propose a dipole source localization (DSL) method that adaptively estimates the dipole number by using a novel information criterion. Since the particle filtering process is nonparametric, it is not clear whether conventional information criteria such as Akaike's information criterion (AIC) and Bayesian information criterion (BIC) can be applied. In the proposed method, multiple particle filters run in parallel, each of which respectively estimates the dipole locations and moments, with the assumption that the dipole number is known and fixed; at every time step, the most predictive particle filter is selected by using an information criterion tailored for particle filters. We tested the proposed information criterion first through experiments on artificial datasets; these experiments supported the hypothesis that the proposed information criterion would outperform both AIC and BIC. We then analyzed real human EEG datasets collected during an auditory short-term memory task using the proposed method. We found that the alpha-band dipoles were localized to the right and left auditory areas during the auditory short-term memory task, which is consistent with previous physiological findings. These analyses suggest the proposed information criterion can work well in both model and real-world situations.

The introduction of photovoltaic power systems is being significantly promoted. This paper proposes the implementation of a distributed energy management framework linking demand-side management systems and supply-side management system under the given time-of-use pricing program for efficient utilization of photovoltaic power outputs; each system implements a consistent management flow composed of forecasting, operation planning, and control steps. In our framework, demand-side systems distributed in the electric distribution network manage individual energy consumption to reduce the residential operating cost by utilizing the residential photovoltaic power system and controllable energy appliances so as not to inconvenience residents. On the other hand, the supply-side system utilizes photovoltaic power maximally while maintaining the quality of electric power. The effectiveness of the proposed framework is evaluated on the basis of an actual Japanese distribution network simulation model from both the supply-side and demand-side viewpoints.

Modal linear regression (MLR) is a standard method for modeling the conditional mode of a response variable using a linear combination of explanatory variables. It is effective when dealing with response variables with an asymmetric, multi-modal distribution. Because of the nonparametric nature of MLR, it is difficult to construct a statistical model manifold in the sense of information geometry. In this work, a model manifold is constructed using observations instead of explicit parametric models. We also propose a method for constructing a data manifold based on an empirical distribution. The em algorithm, which is a geometric formulation of the EM algorithm, of MLR is shown to be equivalent to the conventional EM algorithm of MLR.

Nonnegative matrix factorization (NMF) has many applications as a tool for dimension reduction. In this paper, we reformulate the NMF from an information geometrical viewpoint. We show that a conventional optimization criterion is not geometrically natural, thus we propose to use more natural criterion. By this formulation, we can apply a geometrical algorithm based on the Pythagorean theorem. We also show the algorithm can improve the existing algorithm through numerical experiments.

This paper presents an investigation of the approximation property of neural networks with unbounded activation functions, such as the rectified linear unit (ReLU), which is the new de-facto standard of deep learning. The ReLU network can be analyzed by the ridgelet transform with respect to Lizorkin distributions. By showing three reconstruction formulas by using the Fourier slice theorem, the Radon transform, and Parseval's relation, it is shown that a neural network with unbounded activation functions still satisfies the universal approximation property. As an additional consequence, the ridgelet transform, or the backprojection filter in the Radon domain, is what the network learns after backpropagation. Subject to a constructive admissibility condition, the trained network can be obtained by simply discretizing the ridgelet transform, without backpropagation. Numerical examples not only support the consistency of the admissibility condition but also imply that some non-admissible cases result in low-pass filtering. (C) 2015 Elsevier Inc. All rights reserved.

We propose a method for intrinsic dimension estimation. By fitting the power of distance from an inspection point and the number of samples included inside a ball with a radius equal to the distance, to a regression model, we estimate the goodness of fit. Then, by using the maximum likelihood method, we estimate the local intrinsic dimension around the inspection point. The proposed method is shown to be comparable to conventional methods in global intrinsic dimension estimation experiments. Furthermore, we experimentally show that the proposed method outperforms a conventional local dimension estimation method.

We propose a method for intrinsic dimension estimation. By fitting the power of distance from an inspection point and the number of samples included inside a ball with a radius equal to the distance, to a regression model, we estimate the goodness of fit. Then, by using the maximum likelihood method, we estimate the local intrinsic dimension around the inspection point. The proposed method is shown to be comparable to conventional methods in global intrinsic dimension estimation experiments. Furthermore, we experimentally show that the proposed method outperforms a conventional local dimension estimation method.

A number of image super resolution algorithms based on the sparse coding have successfully implemented multi-frame super resolution in recent years. In order to utilize multiple low-resolution observations, both accurate image registration and sparse coding are required. Previous study on multi-frame super resolution based on sparse coding firstly apply block matching for image registration, followed by sparse coding to enhance the image resolution. In this paper, these two problems are solved by optimizing a single objective function. The proposed formulation not only has a mathematically interesting structure, called the double sparsity, but also yields comparable or improved numerical performance to conventional methods. (C) 2017 Elsevier B.V. All rights reserved.

In a product market or stock market, different products or stocks compete for the same consumers or purchasers. We propose a method to estimate the time-varying transition matrix of the product share using a multivariate time series of the product share. The method is based on the assumption that each of the observed time series of shares is a stationary distribution of the underlying Markov processes characterized by transition probability matrices. We estimate transition probability matrices for every observation under natural assumptions. We demonstrate, on a real-world dataset of the share of automobiles, that the proposed method can find intrinsic transition of shares. The resulting transition matrices reveal interesting phenomena, for example, the change in flows between TOYOTA group and GM group for the fiscal year where TOYOTA group's sales beat GM's sales, which is a reasonable scenario.

This study considers the common situation in data analysis when there are few observations of the distribution of interest or the target distribution, while abundant observations are available from auxiliary distributions. In this situation, it is natural to compensate for the lack of data from the target distribution by using data sets from these auxiliary distributionsin other words, approximating the target distribution in a subspace spanned by a set of auxiliary distributions. Mixture modeling is one of the simplest ways to integrate information from the target and auxiliary distributions in order to express the target distribution as accurately as possible. There are two typical mixtures in the context of information geometry: the The is applied in a variety of research fields because of the presence of the well-known expectation-maximazation algorithm for parameter estimation, whereas the e-mixture is rarely used because of its difficulty of estimation, particularly for nonparametric models. The e-mixture, however, is a well-tempered distribution that satisfies the principle of maximum entropy. To model a target distribution with scarce observations accurately, this letter proposes a novel framework for a nonparametric modeling of the e-mixture and a geometrically inspired estimation algorithm. As numerical examples of the proposed framework, a transfer learning setup is considered. The experimental results show that this framework works well for three types of synthetic data sets, as well as an EEG real-world data set.

Background: Knowledge about the distribution, strength, and direction of synaptic connections within neuronal networks are crucial for understanding brain function. Electrophysiology using multiple electrodes provides a very high temporal resolution, but does not yield sufficient spatial information for resolving neuronal connection topology. Optical recording techniques using single-cell resolution have provided promise for providing spatial information. Although calcium imaging from hundreds of neurons has provided a novel view of the neural connections within the network, the kinetics of calcium responses are not fast enough to resolve each action potential event with high fidelity. Therefore, it is not possible to detect the direction of neuronal connections.
New method: We took advantage of the fast kinetics and large dynamic range of the DiO/DPA combination of voltage sensitive dye and the fast scan speed of a custom-made random-access two-photon microscope to resolve each action potential event from multiple neurons in culture.
Results: Long-duration recording up to 100 min from cultured hippocampal neurons yielded sufficient numbers of spike events for analyzing synaptic connections. Cross-correlation analysis of neuron pairs clearly distinguished synaptically connected neuron pairs with the connection direction.
Comparison with existing method: The long duration recording of action potentials with voltage-sensitive dye utilized in the present study is much longer than in previous studies. Simultaneous optical voltage and calcium measurements revealed that voltage-sensitive dye is able to detect firing events more reliably than calcium indicators.
Conclusions: This novel method reveals a new view of the functional structure of neuronal networks. (C) 2016 Elsevier B.V. All rights reserved.

Unexpected fluctuation of wind power output will become serious problems from the viewpoint of stable supply for an electricity grid. Operating a battery system installed in the grid for mitigating the short-term fluctuation is one of the new approaches for grid stabilization. In this paper, we propose a method of generating synthetic wind power profiles with high temporal resolution for power flow simulation which aims to estimate the impact of wind power fluctuation and specify the required battery system. We numerically show the plausibility of the synthetic wind power profiles from the viewpoints of statistical properties. (C) 2016 The Authors. Published by Elsevier Ltd.

Mixture modeling is one of the simplest ways to represent complicated probability density functions, and to integrate information from different sources. There are two typical mixtures in the context of information geometry, the m- and e-mixtures. This paper proposes a novel framework of non-parametric e-mixture modeling by using a simple estimation algorithm based on geometrical insights into the characteristics of the e-mixture. An experimental result supports the proposed framework.

A method for botnet detection from traffic data of the Internet by the Non-negative Matrix Factorization (NMF) was proposed by (Yamauchi et al. 2012). This method assumes that traffic data is composed by several types of communications, and estimates the number of types in the data by the minimum description length (MDL) criterion. However, consideration on the MDL criterion was not sufficient and validity has not been guaranteed. In this paper, we refine the MDL criterion for NMF and report results of experiments for the new MDL criterion on synthetic and real data.

We propose a method to extract connectivity between neurons for extracellularly recorded multiple spike trains. The method removes pseudo-correlation caused by propagation of information along an indirect pathway, and is also robust against the influence from unobserved neurons. The estimation algorithm consists of iterations of a simple matrix inversion, which is scalable to large data sets. The performance is examined by synthetic spike data.

There are a large number of image super resolution algorithms based on the sparse coding, and some algorithms realize multi-frame super resolution. For utilizing multiple low resolution observations, both accurate image registration and sparse coding are required. Previous study on multi-frame super resolution based on sparse coding firstly apply block matching for image registration, followed by sparse coding to enhance the image resolution. In this paper, these two problems are solved by optimizing a single objective function. The proposed formulation not only has a mathematically interesting structure called the double sparsity, but also offers improved numerical performance.

In this paper, the limitation that is prominent in most existing works of change-point detection methods is addressed by proposing a nonparametric, computationally efficient method. The limitation is that most works assume that each data point observed at each time step is a single multi-dimensional vector. However, there are many situations where this does not hold. Therefore, a setting where each observation is a collection of random variables, which we call a bag of data, is considered.

A method for estimating Shannon differential entropy is proposed based on the second order expansion of the probability mass around the inspection point with respect to the distance from the point. Polynomial regression with Poisson error structure is utilized to estimate the values of density function. The density estimates at every given data points are averaged to obtain entropy estimators. The proposed estimator is shown to perform well through numerical experiments for various probability distributions.

A simple temporal point process (SPP) is an important class of time series, where the sample realization of the process is solely composed of the times at which events occur. Particular examples of point process data are neuronal spike patterns or spike trains, and a large number of distance and similarity metrics for those data have been proposed. A marked point process (MPP) is an extension of a simple temporal point process, in which a certain vector valued mark is associated with each of the temporal points in the SPP. Analyses of MPPs are of practical importance because instances of MPPs include recordings of natural disasters such as earthquakes and tornadoes. In this paper, we introduce the R package mmpp, which implements a number of distance and similarity metrics for SPPs, and also extends those metrics for dealing with MPPs.

In this paper, the limitation that is prominent in most existing works of change-point detection methods is addressed by proposing a nonparametric, computationally efficient method. The limitation is that most works assume that each data point observed at each time step is a single multi-dimensional vector. However, there are many situations where this does not hold. Therefore, a setting where each observation is a collection of random variables, which we call a bag of data, is considered. After estimating the underlying distribution behind each bag of data and embedding those distributions in a metric space, the change-point score is derived by evaluating how the sequence of distributions is fluctuating in the metric space using a distance-based information estimator. Also, a procedure that adaptively determines when to raise alerts is incorporated by calculating the confidence interval of the change-point score at each time step. This avoids raising false alarms in highly noisy situations and enables detecting changes of various magnitudes. A number of experimental studies and numerical examples are provided to demonstrate the generality and the effectiveness of our approach with both synthetic and real datasets.

Estimators for differential entropy are proposed. The estimators are based on the second order expansion of the probability mass around the inspection point with respect to the distance from the point. Simple linear regression is utilized to estimate the values of density function and its second derivative at a point. After estimating the values of the probability density function at each of the given sample points, by taking the empirical average of the negative logarithm of the density estimates, two entropy estimators are derived. Other entropy estimators which directly estimate entropy by linear regression, are also proposed. The proposed four estimators are shown to perform well through numerical experiments for various probability distributions. (C) 2015 Elsevier B.V. All rights reserved.

An image super-resolution method from multiple observation of low-resolution images is proposed. The method is based on sub-pixel accuracy block matching for estimating relative displacements of observed images, and sparse signal representation for estimating the corresponding high-resolution image, where correspondence between high- and low-resolution images are modeled by a certain degradation process. Relative displacements of small patches of observed low-resolution images are accurately estimated by a computationally efficient block matching method. The matching scores of the block matching are used to select a subset of low-resolution patches for reconstructing a high-resolution patch, that is, an adaptive selection of informative low-resolution images is realized. The proposed method is shown to perform comparable or superior to conventional super-resolution methods through experiments using various images. (C) 2015 Elsevier Ltd. All rights reserved.

A classification framework using only a set of distance matrices is proposed. The proposed algorithm can learn a classifier only from a set of distance matrices or similarity matrices, hence applicable to structured data, which do not have natural vector representation such as time series and graphs. Random forest is used to explore ideal feature representation based on the distance between points defined by a set of given distance matrices. The effectiveness of the proposed method is evaluated through experiments with point process data and graph structured data.

An optimal operational planning problem of residential energy system has been formulated by Mixed Integer Linear Programming (MILP). The decision variables of optimal operational planning problem are energy and mass flows, equipment's operating statuses, and energy level of storage. Many kinds of energy supply equipment are available for householders in Japan. Of course, in operational planning problem, the increase of integer variables which means equipment's on/off status, is linked to the increase of calculation time. It is important to assess the impact of introducing an energy system for a house based on the suitable planning horizon of this problem. Energy storage brings the energy in the form of hot water and electricity to the next d a y. The operational strategy of energy system including storage should be evaluated through few days toward various energy demand. The optimal planning problems become large scale because many pieces of equipment to introduce and long evaluation period are required. This paper analyzes characteristics of energy systems caused by planning horizon. Additionally, we propose a hierarchical method as heuristic method for solving large MILP problem easily, and the proposed method is tested the effectiveness. Our finding shows that the proposed search method for better feasible solution has good performance in comparison with default settings of conventional MILP solver in terms of calculation time.

Demand-side electricity saving is an important factor in the reduction of the installed capacity of power-supply facilities. In order to save electricity automatically while maintaining comfort levels, home energy management systems (HEMS) have attracted attention. These systems can control residential energy equipment cooperatively to reduce electricity consumption while considering benefits to consumers. Although many researchers have evaluated HEMS, no one has conducted a study which considers the control of various types of residential energy equipment in real time along with the uncertainties of energy demands. This paper proposes a single HEMS method which connects prediction, operational planning and control steps and enables the evaluation of operational planning methods of HEMS connected with many kinds of residential energy equipment currently in use in Japan while considering the uncertainties. The purpose of this study is to evaluate the economic potential of residential energy systems based on the proposed method with the uncertainties of energy demands and photovoltaic (PV) output under time-of-use prices. The results allowed us to establish a framework to quantitatively evaluate the operational planning methods of HEMS with the uncertainties of energy demands and PV output. In addition, the usability of the proposed method was confirmed by comparing the operational costs to those of a reference method.

We propose methods of estimating the convergence point for the moving vectors of individuals between generations or evolution paths and show that the estimated convergence point can be useful information for accelerating evolutionary computation (EC). As the first stage of this new approach, we do not combine the proposed methods with EC search in this paper, but rather evaluate how power an individual the the estimated convergence point is by comparing fitness values. Through experimental evaluations, we show that the estimated point can be a powerful elite for unimodal fitness landscapes and that clustering moving vectors according to the aimed points is the next research target for multimodal fitness landscape.

Residential houses are in the process of introducing power generators such as photovoltaic (PV) power generators and fuel cell cogeneration systems. Under current laws in Japan, surplus electricity from a residential PV system can be sold by feeding it back into the electrical grid. However, when a lot of neighboring power generators make and feed back electricity at the same time, there is an issue of an upper voltage violation of the provisions of the laws and regulations relating to the Electricity Business Act in the distribution system of the electrical grid. The issue has been solved by stopping power generators when they reach an upper voltage limit, 107V. Another solution is to acknowledge demand response signal from electrical grid operator to store electricity in residential batteries using Home Energy Management System (HEMS). The research question is that how to collaborate HEMS and Grid Energy Management System (GEMS). This paper developed an evaluation framework for the cooperative behavior between HEMS and GEMS. Using this evaluation framework, this paper demonstrated the optimal operational strategy of HEMS including PV in the case that HEMS is informed voltage profile from GEMS, and also assessed amount of PV suppression quantitatively in residential sector in practical aspect.

Residential houses are in the process of introducing power generators such as photovoltaic (PV) power generators and fuel cell cogeneration systems. Under current laws in Japan, surplus electricity from a residential PV system can be sold by feeding it back into the electrical grid. However, when a lot of neighboring power generators make and feed back electricity at the same time, there is an issue of an upper voltage violation of the provisions of the laws and regulations relating to the Electricity Business Act in the distribution system of the electrical grid. The issue has been solved by stopping power generators when they reach an upper voltage limit, 107V. Another solution is to acknowledge demand response signal from electrical grid operator to store electricity in residential batteries using Home Energy Management System (HEMS). The research question is that how to collaborate HEMS and Grid Energy Management System (GEMS). This paper developed an evaluation framework for the cooperative behavior between HEMS and GEMS. Using this evaluation framework, this paper demonstrated the optimal operational strategy of HEMS including PV in the case that HEMS is informed voltage profile from GEMS, and also assessed amount of PV suppression quantitatively in residential sector in practical aspect.

Stable operation of a continuous casting process requires precise control of molten steel temperature in a tundish (TD temp), which is a container used to feed molten steel into an ingot mold. Since TD temp is implicitly controlled by adjusting molten steel temperature in the preceding secondary refining process (RH temp), a model relating TD temp with RH temp is required. This research proposes a procedure to predict the probability distribution of TD temp by integrating a gray-box model and a bootstrap filter to cope with uncertainties of the process. The derived probability distribution is used not only to predict TD temp but also to evaluate the reliability of prediction. The proposed method was validated through its application to real operation data at a steel work, and it was confirmed that the developed model satisfied the requirements for its industrial application.

Clustering is a representative of unsupervised learning and one of the important approaches in exploratory data analysis. By its very nature, clustering without strong assumption on data distribution is desirable. Information-theoretic clustering is a class of clustering methods that optimize information-theoretic quantities such as entropy and mutual information. These quantities can be estimated in a nonparametric manner, and information-theoretic clustering algorithms are capable of capturing various intrinsic data structures. It is also possible to estimate information-theoretic quantities using a data set with sampling weight for each datum. Assuming the data set is sampled from a certain cluster and assigning different sampling weights depending on the clusters, the cluster-conditional information-theoretic quantities are estimated. In this letter, a simple iterative clustering algorithm is proposed based on a nonparametric estimator of the log likelihood for weighted data sets. The clustering algorithm is also derived from the principle of conditional entropy minimization with maximum entropy regularization. The proposed algorithm does not contain a tuning parameter. The algorithm is experimentally shown to be comparable to or outperform conventional nonparametric clustering methods.

A graph is a mathematical representation of a set of variables where some pairs of the variables are connected by edges. Common examples of graphs are railroads, the Internet, and neural networks. It is both theoretically and practically important to estimate the intensity of direct connections between variables. In this study, a problem of estimating the intrinsic graph structure from observed data is considered. The observed data in this study are a matrix with elements representing dependency between nodes in the graph. The dependency represents more than direct connections because it includes influences of various paths. For example, each element of the observed matrix represents a co-occurrence of events at two nodes or a correlation of variables corresponding to two nodes. In this setting, spurious correlations make the estimation of direct connection difficult. To alleviate this difficulty, a digraph Laplacian is used for characterizing a graph. A generative model of this observed matrix is proposed, and a parameter estimation algorithm for the model is also introduced. The notable advantage of the proposed method is its ability to deal with directed graphs, while conventional graph structure estimation methods such as covariance selections are applicable only to undirected graphs. The algorithm is experimentally shown to be able to identify the intrinsic graph structure.

To realize stable production in the steel industry, it is important to control molten steel temperature in a continuous casting process. The present work aims to provide a general framework of gray-box modeling and to develop a gray-box model that predicts and controls molten steel temperature in a tundish (TD temp) with high accuracy. Since the adopted first-principle model (physical model) cannot accurately describe uncertainties such as degradation of ladles, their overall heat transfer coefficient, which is a parameter in the first-principle model, is optimized for each past batch separately, then the parameter is modeled as a function of process variables through a statistical modeling method, random forests. Such a model is termed as a serial gray-box model. Prediction errors of the first-principle model or the serial gray-box model can be compensated by using another statistical model; this approach derives a parallel gray-box model or a combined gray-box model. In addition, the developed gray-box models are used to determine the optimal molten steel temperature in the Ruhrstahl-Heraeus degassing process from the target TD temp, since the continuous casting process has no manipulated variable to directly control TD temp. The proposed modeling and control strategy is validated through its application to real operation data at a steel work. The results show that the combined gray-box model achieves the best performance in prediction and control of TD temp and satisfies the requirement for its industrial application. (C) 2014 Elsevier Ltd. All rights reserved.

Forecast of the energy demand is an important topic for the realization of effective energy management. In this paper, the authors focus on the K-nearest neighbor approach for forecast of the energy demand pattern, introduce an idea of distance metric learning to select appropriate K-nearest neighbors, and propose some learning frameworks for forecasting. The proposed frameworks are evaluated based on the real-world datasets from the viewpoint of selection accuracy of the true neighbors.

A new sampling learning method for neural networks is proposed. Derived from an integral representation of neural networks, an oracle probability distribution of hidden parameters is introduced. In general rigorous sampling from the oracle distribution holds numerical difficulty, a linear-time sampling algorithm is also developed. Numerical experiments showed that when hidden parameters were initialized by the oracle distribution, following backpropagation converged faster to better parameters than when parameters were initialized by a normal distribution. © 2014 Springer International Publishing Switzerland.

A problem of estimating the intrinsic graph structure from observed data is considered. The observed data in this study is a matrix with elements representing dependency between nodes in the graph. Each element of the observed matrix represents, for example, co-occurrence of events at two nodes, or correlation of variables corresponding to two nodes. The dependency does not represent direct connections and includes influences of various paths, and spurious correlations make the estimation of direct connection difficult. To alleviate this difficulty, digraph Laplacian is used for characterizing a graph. A generative model of an observed matrix is proposed, and a parameter estimation algorithm for the model is also proposed. The proposed method is capable of dealing with directed graphs, while conventional graph structure estimation methods from an observed matrix are only applicable to undirected graphs. Experimental result shows that the proposed algorithm is able to identify the intrinsic graph structure. © 2014 Springer International Publishing Switzerland.

A problem of estimating the intrinsic graph structure from observed data is considered. The observed data in this study is a matrix with elements representing dependency between nodes in the graph. Each element of the observed matrix represents, for example, co-occurrence of events at two nodes, or correlation of variables corresponding to two nodes. The dependency does not represent direct connections and includes influences of various paths, and spurious correlations make the estimation of direct connection difficult. To alleviate this difficulty, digraph Laplacian is used for characterizing a graph. A generative model of an observed matrix is proposed, and a parameter estimation algorithm for the model is also proposed. The proposed method is capable of dealing with directed graphs, while conventional graph structure estimation methods from an observed matrix are only applicable to undirected graphs. Experimental result shows that the proposed algorithm is able to identify the intrinsic graph structure. © 2014 Springer International Publishing Switzerland.

Clustering is a fundamental tool for exploratory data analysis. Information theoretic clustering is based on the optimization of information theoretic quantities such as entropy and mutual information. Recently, since these quantities can be estimated in non-parametric manner, non-parametric information theoretic clustering gains much attention. Assuming the dataset is sampled from a certain cluster, and assigning different sampling weights depending on the clusters, the cluster conditional information theoretic quantities are estimated. In this paper, a simple clustering algorithm is proposed based on the principle of maximum entropy. The algorithm is experimentally shown to be comparable to or outperform conventional non-parametric clustering methods. © 2014 Springer International Publishing Switzerland.

Clustering is a fundamental tool for exploratory data analysis. Information theoretic clustering is based on the optimization of information theoretic quantities such as entropy and mutual information. Recently, since these quantities can be estimated in non-parametric manner, non-parametric information theoretic clustering gains much attention. Assuming the dataset is sampled from a certain cluster, and assigning different sampling weights depending on the clusters, the cluster conditional information theoretic quantities are estimated. In this paper, a simple clustering algorithm is proposed based on the principle of maximum entropy. The algorithm is experimentally shown to be comparable to or outperform conventional non-parametric clustering methods. © 2014 Springer International Publishing Switzerland.

The Shannon information content is a valuable numerical characteristic of probability distributions. The problem of estimating the information content from an observed dataset is very important in the fields of statistics, information theory, and machine learning. The contribution of the present paper is in proposing information estimators, and showing some of their applications. When the given data are associated with weights, each datum contributes differently to the empirical average of statistics. The proposed estimators can deal with this kind of weighted data. Similar to other conventional methods, the proposed information estimator contains a parameter to be tuned, and is computationally expensive. To overcome these problems, the proposed estimator is further modified so that it is more computationally efficient and has no tuning parameter. The proposed methods are also extended so as to estimate the cross-entropy, entropy, and Kullback-Leibler divergence. Simple numerical experiments show that the information estimators work properly. Then, the estimators are applied to two specific problems, distribution-preserving data compression, and weight optimization for ensemble regression. (C) 2013 Elsevier Ltd. All rights reserved.

Sparse signal models have been the focus of recent research. In sparse coding, signals are represented with a linear combination of a small number of elementary signals called atoms, and the collection of atoms is called a dictionary. Design of the dictionary has strong influence on the signal approximation performance. Recently, to put prior information into dictionary learning, several methods imposing a certain kind of structure on the dictionary are proposed. In this paper, like wavelet analysis, a dictionary for sparse signal representation is assumed to be generated from an ancestral atom, and a method for learning the ancestral atom is proposed. The proposed algorithm updates the ancestral atom by iterating dictionary update in unstructured dictionary space and projection of the updated dictionary onto the structured dictionary space. The algorithm allows a simple differential geometric interpretation. Numerical experiments are performed to show the characteristics and advantages of the proposed algorithm.

Data clustering is a fundamental technique in many fields of information processing including image analysis. The results of data clustering depend on both the clustering algorithm and the similarity defined between pair of the data. Focusing the distribution of the data around a line connecting a pair of data points, a method of defining a similarity between the pair of data is proposed. The similarity is used for clustering the observed dataset. The method is applied to detecting line segments from distorted images taken by cameras with a fish-eye lens.

Remarkable progress has been made in genome science during the past decade, but understanding of genomes of eukaryotes is far from complete. We have created DNA flexibility maps of the human, mouse, fruit fly, and nematode chromosomes. The maps revealed that all of these chromosomes have markedly flexible DNA regions (We named them SPIKEs). SPIKEs occur more frequently in the human chromosomes than in the mouse, fruit fly, and nematode chromosomes. Markedly rigid DNA regions (rSPIKEs) are also present in these chromosomes. The ratio of the number of SPIKEs to the total number of SPIKEs and rSPIKEs correlated positively with evolutionary stage among the organisms. Repetitive DNA sequences with flexible and rigid properties contribute to the formation of SPIKEs and rSPIKEs respectively. However, non-repetitive flexible and rigid sequences appear to play a major role in SPIKE and rSPIKE formation respectively. They might be involved in the genome-folding mechanism of eukaryotes.

Clustering a given set of data is crucial in many fields including image processing. It plays important roles in image segmentation and object detection for example. This paper proposes a framework of building a similarity matrix for a given dataset, which is then used for clustering the dataset. The similarity between two points are defined based on how other points distribute around the line connecting the two points. It can capture the degree of how the two points are placed on the same line. The similarity matrix is considered as a kernel matrix of the given dataset, and based on it, the spectral clustering is performed. Clustering with the proposed similarity matrix is shown to perform well through experiments using an artificially designed problem and a real-world problem of detecting lines from a distorted image. © 2013 Springer-Verlag.

A novel gray-box model is proposed to estimate molten steel temperature in a continuous casting process at a steel making plant by combining a first-principle model and a statistical model. The first-principle model was developed on the basis of computational fluid dynamics (CFD) simulations to simplify the model and to improve estimation accuracy. Since the derived first-principle model was not able to estimate the molten steel temperature in the tundish with sufficient accuracy, statistical models were developed to estimate the estimation errors of the first-principle model through partial least squares (PLS) and random forest (RF). As a result of comparing the three models, i.e., the first-principle model, the PLS-based gray-box model, and the RF-based gray-box model, the RF-based gray-box model achieved the best estimation performance. Thus, the molten steel temperature in the tundish can be estimated with accuracy by adding estimates of the first-principle model and those of the statistical RF model. The proposed gray-box model was applied to the real process data and the results demonstrated its advantage over other models.

Abstract The importance of dimension reduction has been increasing according to the growth of the size of available data in many fields. An appropriate dimension reduction method of raw data helps to reduce computational time and to expose the intrinsic structure of complex data. Sliced inverse regression is a well-known dimension reduction method for regression, which assumes an elliptical distribution for the explanatory variable, and ingeniously reduces the problem of dimension reduction to a simple eigenvalue problem. Sliced inverse regression is based on the strong assumptions on the data distribution and the form of regression function, and there are a number of methods to relax or remove these assumptions to extend the applicability of the inverse regression method. However, each method is known to have its drawbacks either theoretically or empirically. To alleviate drawbacks in the existing methods, a dimension reduction method for regression based on the notion of conditional entropy minimization is proposed. Using entropy as a measure of dispersion of data, a low dimensional subspace is estimated without assuming any specific distribution nor any regression function. The proposed method is shown to perform comparable or superior to the conventional methods through experiments using artificial and real-world datasets. © 2013 Elsevier B.V. All rights reserved.

In order to realize stable production in the steel industry, it is important to control molten steel temperature in a continuous casting process. The present work aims to develop a gray-box model that predicts the molten steel temperature in the tundish (TD temp). In the proposed approach, two parameters in the first-principle model, i.e., overall heat transfer coefficients of ladle and tundish, are optimized for each past batch separately, then the relationship between the two parameters and measured process variables is modeled through random forests (RF). In this inner gray-box model, the statistical models update the physical parameters according to the operating condition. To enhance the accuracy of TD temp estimation, another RF model is developed which compensates errors of the inner gray-box. The proposed approach was validated through its application to real operation data at a steel work. © IFAC.

Analysis and modeling of electric energy demand is indispensable for power planning, operation, facility investment, and urban planning. Because of recent development of renewable energy generation systems oriented for households, there is also a great demand for analysing the electricity usage and optimizing the way to install electricity generation systems for each household. In this study, employing statistical techniques, a method to model daily consumption patterns in households and a method to extract a small number of their typical patterns are presented. The electricity consumption patterns in a household is modeled by a mixture of Gaussian distributions. Then, using the symmetrized generalized Kullback-Leibler divergence as a distance measure of the distributions, typical patterns of the consumption are extracted by means of hierarchical clustering. The statistical modeling of daily consumption patterns allows us to capture essential similarities of the patterns. By experiments using a large-scale dataset including about 500 houses' consumption records in a suburban area in Japan, it is shown that the proposed method is able to extract typical consumption patterns. © 2010-2012 IEEE.

When evaluating residential energy systems like co-generation systems, hot water and electricity demand profiles are critical. In this paper, the authors aim to extract basic time-series demand patterns from two kinds of measured demand (electricity and domestic hot water), and also aim to reveal effective demand patterns for primary energy saving. Time-series demand data are categorized with a hierarchical clustering method using a statistical pseudo-distance, which is represented by the generalized Kullback-Leibler divergence of two Gaussian mixture distributions. The classified demand patterns are built using hierarchical clustering and then a comparison is made between the optimal operation of a polymer electrolyte membrane fuel cell co-generation system and the operation of a reference system (a conventional combination of a condensing gas boiler and electricity purchased from the grid) using the appropriately built demand profiles. Our results show that basic demand patterns are extracted by the proposed method, and the heat-to-power ratio of demand, the amount of daily demand, and demand patterns affect the primary energy saving of the co-generation system. © 2013 by the authors
licensee MDPI, Basel, Switzerland.

Kernel methods are known to be effective for nonlinear multivariate analysis. One of the main issues in the practical use of kernel methods is the selection of kernel. There have been a lot of studies on kernel selection and kernel learning. Multiple kernel learning (MKL) is one of the promising kernel optimization approaches. Kernel methods are applied to various classifiers including Fisher discriminant analysis (FDA). FDA gives the Bayes optimal classification axis if the data distribution of each class in the feature space is a gaussian with a shared covariance structure. Based on this fact, an MKL framework based on the notion of gaussianity is proposed. As a concrete implementation, an empirical characteristic function is adopted to measure gaussianity in the feature space associated with a convex combination of kernel functions, and two MKL algorithms are derived. From experimental results on some data sets, we show that the proposed kernel learning followed by FDA offers strong classification power.

An appropriate dimension reduction of raw data helps to reduce computational time and to reveal the intrinsic structure of complex data. In this paper, a dimension reduction method for regression is proposed. The method is based on the well-known sliced inverse regression and conditional entropy minimization. Using entropy as a measure of dispersion of data distribution, dimension reduction subspace is estimated without assuming regression function form nor data distribution, unlike conventional sliced inverse regression. The proposed method is shown to perform well compared to some conventional methods through experiments using both artificial and real-world data sets.

Controlling temperature of molten steel is crucial for product quality in continuous casting. In this paper, sensitivity analysis is carried out on a statistical model for predicting temperature in tundish, and influential operations for controlling temperature are identified. © 2012 IFAC.

This paper considers N -1-dimensional hypersurface fitting based on L-2 distance in N-dimensional input space. The problem is usually reduced to hyperplane fitting in higher dimension. However, because feature mapping is generally a nonlinear mapping, it does not preserve the order of lengthes, and this derives an unacceptable fitting result. To avoid it, JNLPCA is introduced. JNLPCA defines the L-2 distance in the feature space as a weighted L-2 distance to reflect the metric in the input space. In the fitting, random sampling approximation of least k-th power deviation, and least alpha-percentile of squares are introduced to make estimation robust. The proposed hypersurface fitting method is evaluated by quadratic curve fitting and quadratic curve segments extraction from artificial data and a real image.

Nonnegative Matrix Factorization (NMF) is broadly used as a mathematical tool for processing tasks of tabulated data. In this paper, an extension of NMF based on a generalized product rule, defined with a nonlinear one-parameter function and its inverse, is proposed. From a viewpoint of subspace methods, the extended NMF constructs flexible subspaces which plays an important role in classification tasks. Experimental results on benchmark datasets show that the proposed extension improves classification accuracies. © 2012 Springer-Verlag.

Controlling molten steel temperature in a tundish is crucial for stable and efficient production of steel products. In this research, a gray-box model is developed by combining a first-principle model and a statistical model. The first-principle model predicts the molten steel temperature in the tundish, and the statistical model predicts the prediction error of the first-principle model. The derived model was then used to derive the molten steel temperature in the Ruhrstahl-Heraeus process at the end of its operation from the target molten steel temperature in the tundish and other process variables. The result of applying the proposed model to real operation data has demonstrated its practicability. © 2012 IFAC.

Electricity consumption in households varies dependent on a lot of possible reasons such as lifestyle, family configuration, and weather. It is of great importance to optimize the electricity generation system to install for each household. In our previous work, we proposed a clustering approach for extracting a small number of basic electricity consumption patterns in a household. In this study, we apply the method to a larger dataset with many households. In the previous work, we determined the number of basic patterns in a heuristic manner. In this work, we use gap statistics to automatically determine an appropriate number of basic patterns, and we obtained a reasonable result on a large-scale data.

Sparse coding is an important optimization problem with numerous applications. In this paper, we describe the problem and the commonly used pursuit methods, and propose a best-first tree search algorithm employing multiple queues for unexplored tree nodes. We assess the effectiveness of our method in an extensive computational experiment, showing its superiority over other methods even for modest computational time. © 2012 Springer-Verlag.

A statistical model for predicting the liquid steel temperature in the ladle and in the tundish is developed. Given a large data set in a steelmaking process, the proposed model predicts the temperature in a seconds with a good accuracy. The data are divided into four phases at the mediation of five temperature measurements: before tapping from the converter (CV), after throwing ferroalloys into the ladle, before and after the Ruhrstahl-Heraeus (RH) processing, and after casting into the tundish in the continuous casting (CC) machine. Based on the general state space modeling, the bootstrap filter predicts the temperature phase by phase. The particle approximation technique enables to compute general-shaped probability distributions. The proposed model gives a prediction not as a point but as a probability distribution, or a predictive distribution. It evaluates both uncertainty of the prediction and ununiformity of the temperature. It is applicable to sensitivity analysis, process scheduling and temperature control.

In this paper, generalised statistical independence in statistical models for categorical distributions is proposed from the viewpoint of generalised multiplication characterised by a monotonically increasing function and its inverse function, and it is implemented in naive Bayes models. This paper also proposes an idea of their estimation method which directly uses empirical marginal distributions to retain simplicity of calculation. This method is interpreted as an optimisation of a rough approximation of the Bregman divergence so that it is expected to have a kind of robust property. Effectiveness of proposed models is shown by numerical experiments on some benchmark datasets. Copyright © 2012 Inderscience Enterprises Ltd.

Residential energy demand varies widely in terms of time-series behaviors, amounts consumed between families, and even within one family. Residential energy demand profiles have a high degree of uncertainty in their essentials because the demand profile is entirely based on the occupant-driven load. When evaluating residential energy systems like co-generation systems, hot water and electricity demand profiles are critical. In this paper, in order to clarify rational energy system selection guidelines and rational operation strategies, authors aim to extract basic demand time-series patterns from two kinds of measured demand (electricity and domestic hot water), measured over 26307 days of data in Japan. Authors also aim to reveal the relationship between primary energy consumption and demand patterns. Demand time-series data are categorized by means of a kind of "unsupervised" learning, which is a hierarchical clustering method using a statistical pseudo-distance. The statistical pseudo-distance is calculated from the generalized Kullback-Leibler divergence with the Gaussian mixture distribution fitted to the demand time-series data. The classified demand patterns are built using a hierarchical clustering and then a comparison is performed between the optimal operation of the two systems (a polymer electrolyte membrane fuel cell co-generation system, and a CO2 heat pump system) and the operation of a reference system (a conventional combination of a condensing gas boiler and electricity purchased from the grid) using the demand profiles appropriately built. Our results show that basic demand patterns are extracted by the proposed method. The demand patterns, the amount of daily demand and heat-to-power ratio of demand affect the primary energy reduction ratio of the polymer electrolyte membrane fuel cell co-generation system.

The Bradley-Terry model is a statistical representation for one's preference or ranking data by using pairwise comparison results of items. For estimation of the model, several methods based on the sum of weighted Kullback-Leibler divergences have been proposed from various contexts. The purpose of this letter is to interpret an estimation mechanism of the Bradley-Terry model from the viewpoint of flatness, a fundamental notion used in information geometry. Based on this point of view, a new estimation method is proposed on a framework of the em algorithm. The proposed method is different in its objective function from that of conventional methods, especially in treating unobserved comparisons, and it is consistently interpreted in a probability simplex. An estimation method with weight adaptation is also proposed from a viewpoint of the sensitivity. Experimental results show that the proposed method works appropriately, and weight adaptation improves accuracy of the estimate.

We developed a new speaker verification system that is robust to intra-speaker variation. There is a strong likelihood that intra-speaker variations will occur due to changes in talking styles, the periods when an individual speaks, and so on. It is well known that such variation generally degrades the performance of speaker verification systems. To solve this problem, we applied multiple kernel learning (MKL) based on conditional entropy minimization, which impose the data to be compactly aggregated for each speaker class and ensure that the different speaker classes were far apart from each other. Experimental results showed that the proposed speaker verification system achieved a robust performance to intra-speaker variation derived from changes in the talking styles compared to the conventional maximum margin-based system.

We applied a multiple kernel learning (MKL) method based on information-theoretic optimization to speaker recognition. Most of the kernel methods applied to speaker recognition systems require a suitable kernel function and its parameters to be determined for a given data set. In contrast, MKL eliminates the need for strict determination of the kernel function and parameters by using a convex combination of element kernels. In the present paper, we describe an MKL algorithm based on conditional entropy minimization (MCEM). We experimentally verified the effectiveness of MCEM for speaker classification; this method reduced the speaker error rate as compared to conventional methods.

In this paper, two methods for one-dimensional reduction of data by hyperplane fitting are proposed. One is least a-percentile of squares, which is an extension of least median of squares estimation and minimizes the a-percentile of squared Euclidean distance. The other is least k-th power deviation, which is an extension of least squares estimation and minimizes the k-th power deviation of squared Euclidean distance. Especially, for least k-th power deviation of 0 < k <= 1, it is proved that a useful property, called optimal sampling property, holds in one-dimensional reduction of data by hyperplane fitting. The optimal sampling property is that the global optimum for affine hyperplane fitting passes through N data points when an N-1-dimensional hyperplane is fitted to the N-dimensional data. The performance of the proposed methods is evaluated by line fitting to artificial data and a real image.

We derive and demonstrate new methods for dewarping images depicted in convex mirrors in artwork and for estimating the three-dimensional shapes of the mirrors themselves. Previous methods were based on the assumption that mirrors were spherical or paraboloidal, an assumption unlikely to hold for hand-blown glass spheres used in early Renaissance art, such as Johannes van Eyck's Portrait of Giovanni (?) Arnolfini and his wife (1434) and Robert Campin's Portrait of St. John the Baptist and Heinrich von Werl (1438). Our methods are more general than such previous methods in that we assume merely that the mirror is radially symmetric and that there are straight lines (or colinear points) in the actual source scene. We express the mirror's shape as a mathematical series and pose the image dewarping task as that of estimating the coefficients in the series expansion. Central to our method is the plumbline principle: that the optimal coefficients are those that dewarp the mirror image so as to straighten lines that correspond to straight lines in the source scene. We solve for these coefficients algebraically through principal component analysis, PCA. Our method relies on a global figure of merit to balance warping errors throughout the image and it thereby reduces a reliance on the somewhat subjective criterion used in earlier methods. Our estimation can be applied to separate image annuli, which is appropriate if the mirror shape is irregular. Once we have found the optimal image dewarping, we compute the mirror shape by solving a differential equation based on the estimated dewarping function. We demonstrate our methods on the Arnolfini mirror and reveal a dewarped image superior to those found in prior work-an image noticeably more rectilinear throughout and having a more coherent geometrical perspective and vanishing points. Moreover, we find the mirror deviated from spherical and paraboloidal shape; this implies that it would have been useless as a concave projection mirror, as has been claimed. Our dewarped image can be compared to the geometry in the full Arnolfini painting; the geometrical agreement strongly suggests that van Eyck worked from an actual room, not, as has been suggested by some art historians, a "fictive" room of his imagination. We apply our method to other mirrors depicted in art, such as Parmigianino's Self-portrait in a convex mirror and compare our results to those from earlier computer graphics simulations.

Solar power, wind power, and co-generation (combined heat and power) systems are possible candidate for household power generation. These systems have their advantages and disadvantages. To propose the optimal combination of the power generation systems, the extraction of basic patterns of energy consumption of the house is required. In this study, energy consumption patterns are modeled by mixtures of Gaussian distributions. Then, using the symmetrized Kullback-Leibler divergence as a distance measure of the distributions, the basic pattern of energy consumption is extracted by means of hierarchical clustering. By an experiment using the Annex 42 dataset, it is shown that the proposed method is able to extract typical energy consumption patterns. © 2011 IEEE.

For calibration of general radially symmetric distortion of omnidirectional cameras such as fish-eye lenses, calibration parameters are usually estimated so that curved lines, which are supposed to be straight in the real-world, are mapped to straight lines in the calibrated image, which is called plumbline principle. Under the principle, the camera with radially symmetric distortion can be calibrated by at least one distorted line in a image, theoretically, and the calibrated image is equivalent to the image taken by an ideal pin-hole camera. In this paper, the method to optimize the calibration parameters by maximizing the sum of the straightness, which is invariant under translation, rotation and magnification (scaling), of distorted lines on calibrated image is proposed. The performance of the proposed method is evaluated by artificial data and a real image.

Recently, remarkable developments of new energy technologies have been achieved against various energy problems. Photovoltaic (PV) system, one of such technologies, has an advantage of utilizing infinite and clean energy. On the contrary, it also has a disadvantage of unreliable power supply mainly caused by unstable weather. The fluctuation of the power supply of PV systems are considerably large because of rapid insulation changes and rapid weather changes, and in some cases, it seems impossible to realize high-accuracy prediction even with sophisticated prediction models. In this paper, using recently proposed estimator for the Shannon information content, a method to output a measure of credibility for prediction is proposed. With the proposed method, it is possible to judge whether the energy supply at a certain future time is unpredictably fluctuate compared to the current value or not, and it is possible to take measures against the rapid change of solar energy generation in advance. From an experimental result using solar energy supply data, we see that the proposed measure of credibility reflects the difficulty of predicting solar energy supply. © 2011 IEEE.

The Shannon information content is a fundamental quantity and it is of great importance to estimate it from observed dataset in the field of statistics, information theory, and machine learning. In this study, an estimator for the information content using a given set of weighted data is proposed. The empirical data distribution varies depending on the weight. The notable features of the proposed estimator are its computational efficiency and its ability to deal with weighted data. The proposed estimator is extended in order to estimate cross entropy, entropy and KL divergence with weighted data. Then, the estimators are applied to classification with one-class samples, and distribution preserving data compression problems.

Molecular events in biological cells occur in local subregions, where the molecules tend to be small in number. The cytoskeleton, which is important for both the structural changes of cells and their functions, is also a countable entity because of its long fibrous shape. To simulate the local environment using a computer, stochastic simulations should be run. We herein report a new method of stochastic simulation based on random walk and reaction by the collision of all molecules. The microscopic reaction rate P-r is calculated from the macroscopic rate constant k. The formula involves only local parameters embedded for each molecule. The results of the stochastic simulations of simple second-order, polymerization, Michaelis-Menten-type and other reactions agreed quite well with those of deterministic simulations when the number of molecules was sufficiently large. An analysis of the theory indicated a relationship between variance and the number of molecules in the system, and results of multiple stochastic simulation runs confirmed this relationship. We simulated Ca2+ dynamics in a cell by inward flow from a point on the cell surface and the polymerization of G-actin forming F-actin. Our results showed that this theory and method can be used to simulate spatially inhomogeneous events.

Reducing the dimensionality of high-dimensional data without losing its essential information is an important task in information processing. When class labels of training data are available, Fisher discriminant analysis (FDA) has been widely used. However, the optimality of FDA is guaranteed only in a very restricted ideal circumstance, and it is often observed that FDA does not provide a good classification surface for many real problems. This letter treats the problem of supervised dimensionality reduction from the viewpoint of information theory and proposes a framework of dimensionality reduction based on class-conditional entropy minimization. The proposed linear dimensionality-reduction technique is validated both theoretically and experimentally. Then, through kernel Fisher discriminant analysis (KFDA), the multiple kernel learning problem is treated in the proposed framework, and a novel algorithm, which iteratively optimizes the parameters of the classification function and kernel combination coefficients, is proposed. The algorithm is experimentally shown to be comparable to or outperforms KFDA for large-scale benchmark data sets, and comparable to other multiple kernel learning techniques on the yeast protein function annotation task.

Given a set of rating data for a set of items, determining preference levels of items is a matter of importance. Various probability models have been proposed to solve this task. One such model is the Plackett-Luce model, which parameterizes the preference level of each item by a real value. In this letter, the Plackett-Luce model is generalized to cope with grouped ranking observations such as movie or restaurant ratings. Since it is difficult to maximize the likelihood of the proposed model directly, a feasible approximation is derived, and the em algorithm is adopted to find the model parameter by maximizing the approximate likelihood which is easily evaluated. The proposed model is extended to a mixture model, and two applications are proposed. To show the effectiveness of the proposed model, numerical experiments with real-world data are carried out.

For self-calibration of general radially symmetric distortion (RSD) of omnidirectional cameras such as fish-eye lenses, calibration parameters are usually estimated so that curved lines, which are supposed to be straight in the real-world, are mapped to straight lines in the calibrated image, which is assumed to be taken by an ideal pin-hole camera. In this paper, a method of calibrating RSD is introduced base on the notion of principal component analysis (PCA). In the proposed method, the distortion function, which maps a distorted image to an ideal pin-hole camera image, is assumed to be a linear combination of a certain class of basis functions, and an algorithm for solving its coefficients by using line patterns is given. Then a method of selecting good basis functions is proposed, which aims to realize appropriate calibration in practice. Experimental results for synthetic data and real images are presented to demonstrate the performance of our calibration method. © 2010 IEEE.

Kernel methods have been successfully used in many practical machine learning problems. Choosing a suitable kernel is left to the practitioner. A common way to an automatic selection of optimal kernels is to learn a linear combination of element kernels. In this paper, a novel framework of multiple kernel learning is proposed based on conditional entropy minimization criterion. For the proposed framework, three multiple kernel learning algorithms are derived. The algorithms are experimentally shown to be comparable to or outperform kernel Fisher discriminant analysis and other multiple kernel learning algorithms on benchmark data sets. © 2010 IEEE.

In structure from motion problems, mirror shape reconstruction from mirror image is very interesting. In this paper, we propose a method to estimate the shape of a rotationally symmetric mirror by calibrating a mirror image which has radially symmetric distortion. In the proposed method, the shape of mirror is represented by the set of solution of a differential equation, which is derived from distortion function computed by the calibration of radially symmetric distortion of an image on a rotationally symmetric mirror. The differential equation and the boundary condition are simplified under the assumption that the image on mirror is taken by orthographic projection camera. The experimental result for a real image is presented to demonstrate the performance of our reconstruction method, and the method is applied for reconstruction of the shape of the mirror painted on the Jan van Eyck's Arnolfini Wedding which had drawn in the 15th century. © 2010 IEEE.

In this paper, generalized statistical independence is proposed from the viewpoint of generalized multiplication characterized by a monotonically increasing function and its inverse function, and it is implemented in naive Bayes models. This paper also proposes an idea of their estimation method which directly uses empirical marginal distributions to retain simplicity of calculation. Our method is interpreted as an optimization of a rough approximation of the Bregman divergence so that it is expected to have a kind of robust property. Effectiveness of our proposed models is shown by numerical experiments on some benchmark data sets.

In this chapter, a problem of estimating model parameters from observed data is considered such as regression and function approximation, and a method of evaluating the goodness of model is introduced. Starting from so-called leave-one-out cross-validation, and investigating asymptotic statistical properties of estimated parameters, a generalized Akaike's information criterion (AIC) is derived for selecting an appropriate model from several candidates. In addition to model selection, the concept of information criteria provides an assessment of the goodness of model in various situations. Finally, an optimization method using regularization is presented as an example. © 2009 Springer US.

In this paper, we propose a visualization technique of a statistical relation of users and preference of items based on a mixture model. In our visualization, items are given as points in a few dimensional preference space, and user specific preferences are given as lines in the same space. The relationship between items and user preferences are intuitively interpreted via projections from points onto lines. As a primitive implementation, we introduce a mixture of the Bradley-Terry models, and visualize the relation between items and user preferences with benchmark data sets.

Given a set of rating data for a set of items, determining the values of items is a matter of importance. Various probability models have been proposed to solve this task. The Plackett-Luce model is one of such models, which parametrizes the value of each item by a real valued preference parameter. In this paper, the Plackett-Luce model is generalized to cope with the grouped ranking observations such as movies or restaurants ratings. Since the maximization of the likelihood of the proposed model is computationally intractable, the lower bound of the likelihood which is easy to evaluate is derived, and the ern, algorithm is adopted to the maximization of the lower bound.

To calibrate radially symmetric distortion of omnidirectional cameras such as fish-eye lenses, calibration parameters are usually estimated so drat lines, which are supposed to be straight, in the 3D real scene; are mapped to straight lines in the calibrated image. In this paper, this problem is treated as a fitting problem of the principal component in uncalibrated images, and an estimation procedure of calibration parameters is proposed based oil the principal component analysis. Experimental results for synthetic data and real images are presented to demonstrate the performance of our calibration method.

The sparse coding method is formulated as an information theoretic optimization problem. The rate distortion theory leads to all objective functional which can be interpreted as an information theoretic formulation of the sparse coding. Viewing as an entropy minimization problem, the rate distortion theory and consequently the sparse coding are extended to discriminative variants. As a concrete example of this information theoretic sparse coding, a discriminative non-linear sparse coding algorithm with neural networks is proposed. Experimental results of gender classification by face images show that the discriminative sparse coding is more robust to noise, compared to the conventional method which directly uses images as inputs to a linear support vector machine.

We discuss robustness against mislabeling in multiclass labels for classification problems and propose two algorithms of boosting, the normalized Eta-Boost.M and Eta-Boost.M, based on the Eta-divergence. Those two boosting algorithms are closely related to models of mislabeling in which the label is erroneously exchanged for others. For the two boosting algorithms, theoretical aspects supporting the robustness for mislabeling are explored. We apply the proposed two boosting methods for synthetic and real data sets to investigate the performance of these methods, focusing on robustness, and confirm the validity of the proposed methods.

Previous psychological studies have shown that musical chords primed by Western musical scale in a tonal and modal schema are perceived in a hierarchy of stability. We investigated such priming effects on auditory magnetic responses to tonic-major and submediant-minor chords preceded by major scales and tonic-minor and submediant-major chords preceded by minor scales. Musically trained subjects participated in the experiment. During MEG recordings, subjects judged perceptual stability of the chords. The tonic chords were judged to be stable, whereas the submediant chords were judged to be unstable. Dipole moments of N1m response originating in the auditory cortex were larger in the left hemisphere for the submediant chords than for the tonic chords preceded by the major but not minor scales. No difference in the N1m or P2m moment was found for the chords presented without preceding scales. These results suggest priming effects of the tonal schema, interacting with contextual modality, on neural activity of the auditory cortex as well as perceptual stability of the chords. It is inferred that modulation of the auditory cortical activity is associated with attention induced by tonal instability and modality shift, which characterize the submediant chords. (c) 2007 Elsevier Ireland Ltd and the Japan Neuroscience Society. All rights reserved.

To describe the relation between some values, arithmetic operations like multiplication and division are important and conventional tools. These arithmetic operations for probabilities are characterized by the KL divergence, and hence, they can be generalized by using the Bregman divergence instead of the KL divergence. With this idea, independence of random variables is modified by generalized product rule, and a joint probability model is proposed based on this modified weak independence. Effectiveness of weak independent models is shown by numerical experiments on toy examples, and discussed from a geometrical viewpoint.

Boosting is known as a gradient descent algorithm over loss functions. It is often pointed out that the typical boosting algorithm, Adaboost, is highly affected by outliers. In this letter, loss functions for robust boosting are studied. Based on the concept of robust statistics, we propose a transformation of loss functions that makes boosting algorithms robust against extreme outliers. Next, the truncation of loss functions is applied to contamination models that describe the occurrence of mislabels near decision boundaries. Numerical experiments illustrate that the proposed loss functions derived from the contamination models are useful for handling highly noisy data in comparison with other loss functions.

Boosting is known as a gradient descent algorithm over loss functions. It is often pointed out that the typical boosting algorithm, Adaboost, is highly affected by outliers. In this letter, loss functions for robust boosting are studied. Based on the concept of robust statistics, we propose a transformation of loss functions that makes boosting algorithms robust against extreme outliers. Next, the truncation of loss functions is applied to contamination models that describe the occurrence of mislabels near decision boundaries. Numerical experiments illustrate that the proposed loss functions derived from the contamination models are useful for handling highly noisy data in comparison with other loss functions.

The EM algorithm is a sophisticated method for estimating statistical models with hidden variables based on the Kullback-Leibler divergence. A natural extension of the Kullback-Leibler divergence is given by a class of Bregman divergences, which in general enjoy robustness to contamination data in statistical inference. In this paper, a modification of the EM algorithm based on the Bregman divergence is proposed for estimating finite mixture models. The proposed algorithm is geometrically interpreted as a sequence of projections induced from the Bregman divergence. Since a rigorous algorithm includes a nonlinear optimization procedure, two simplification methods for reducing computational difficulty are also discussed from a geometrical viewpoint. Numerical experiments on a toy problem are carried out to confirm appropriateness of the simplifications.

In this article, several boosting methods are discussed, which are notable implementations of the ensemble learning. Starting from the firstly introduced "boosting by filter" which is an embodiment of the proverb "Two heads are better than one", more advanced versions of boosting methods "AdaBoost" and "U-Boost" are introduced. A geometrical structure and some statistical properties such as consistency and robustness of boosting algorithms are discussed, and then simulation studies are presented for confirming discussed behaviors of algorithms.

Modeling of a large joint probability table is problematic when its variables have a large number of categories. In such a case, a mixture of simpler probability tables could be a good model. And the estimation of such a large probability table frequently has another problem of data sparseness. When constructing mixture models with sparse data, EM estimators based on the beta-likelihood are expected more appropriate than those based on the log likelihood. Experimental results show that a mixture model estimated by the beta-likelihood approximates a large joint probability table with sparse data more appropriately than EM estimators.

By employing the L-1 or L-infinity norms in maximizing margins, support vector machines (SVMs) result in a linear programming problem that requires a lower computational load compared to SVMs with the L-2 norm. However, how the change of norm affects the generalization ability of SVMs has not been clarified so far except for numerical experiments. In this letter, the geometrical meaning of SVMs with the L-p norm is investigated, and the SVM solutions are shown to have rather little dependency on p.

A modified Gaussian process regression is proposed aiming at making regressors robust against outliers. The proposed method is based on U-loss, which is introduced as a natural extension of Kullback-Leibler divergence. The robustness is examined based on the influence function, and numerical experiments are conducted for contaminated data sets and it is shown that the practical performance agrees with the theoretical analysis.

Support Vector Machines (SVMs) are known to have a high generalization ability, yet a heavy computational load since margin maximization results in a quadratic programming problem. It is known that this maximization task results in a pth-order programming problem if we employ the LP norm instead of the L-2 norm. In this paper, we theoretically show the effects of p on the learning properties of SVMs by clarifying its geometrical meaning.

We aim at an extension of AdaBoost to U-Boost, in the paradigm to build a stronger classification machine from a set of weak learning machines. A geometric understanding of the Bregman divergence defined by a generic convex function U leads to the U-Boost method in the framework of information geometry extended to the space of the finite measures over a label set. We propose two versions of U-Boost learning algorithms by taking account of whether the domain is restricted to the space of probability functions. In the sequential step, we observe that the two adjacent and the initial classifiers are associated with a right triangle in the scale via the Bregman divergence, called the Pythagorean relation. This leads to a mild convergence property of the U-Boost algorithm as seen in the expectation-maximization algorithm. Statistical discussions for consistency and robustness elucidate the properties of the U-Boost methods based on a stochastic assumption for training data.

Natural gradient learning is known to be efficient in escaping plateau, which is a main cause of the slow learning speed of neural networks. The adaptive natural gradient learning method for practical implementation also has been developed, and its advantage in real-world problems has been confirmed. In this letter, we deal with the generalization performances of the natural gradient method. Since natural gradient learning makes parameters fit to training data quickly, the overfitting phenomenon may easily occur, which results in poor generalization performance. To solve the problem, we introduce the regularization term in natural gradient learning and propose an efficient optimizing method for the scale of regularization by using a generalized Akaike information criterion (network information criterion). We discuss the properties of the optimized regularization strength by NIC through theoretical analysis as well as computer simulations. We confirm the computational efficiency and generalization performance of the proposed method in real-world applications through computational experiments on benchmark problems.

Boosting algorithm is understood as the gradient descent algorithm of a loss function. It is often pointed out that the typical boosting algorithm, Adaboost, is seriously affected by the outliers. In this paper, loss functions for robust boosting are studied. Based on a concept of the robust statistics, we propose a positive-part-truncation of the loss function which makes the boosting algorithm robust against extreme outliers. Numerical experiments show that the proposed boosting algorithm is useful for highly noisy data in comparison with other competitors.

Recovering both camera motion and object shape from multiple images, called structure from motion problem, is an important and essential problem in computer vision. Generally, the result of the structure from motion problem has an ambiguity represented by a three-dimensional rotation matrix. We present two kinds of specific criteria such as independence of parameters to fix the ambiguity by choosing an appropriate rotation matrix in the sense of computer vision. Once some criterion is defined, the fixing of the ambiguity is reduced to a nonlinear extension of the PCA/ICA. We examine the efficiency through synthetic experiments.

Support Vector Machines (SVMs) are a new classification technique which has a high generalization ability, yet a heavy computational load since margin maximization results in a quadratic programming problem. It is known that this maximization task results in a pth-order programming problem if we employ the p-norm instead of the Euclidean norm, that is. When p = 1, for example, it is a linear programming problem with a much lower computational load. In this article, we theoretically show that p has very little affect on the generalization performance of SVMs in practice by considering its geometrical meaning.

Factor analysis is well known technique to uncorrelate observed signals with Gaussina noises before ICA (Independent Component Analysis) algorithms are applied. However, factor analysis is not applicable when the number of source signals are more than that of Ledermann's bound, and when the observations are contaminated by non-Gaussian noises. In this paper, an approach is proposed based on higher-order moments of signals and noises in order to overcome those constraints.

In this paper, we propose a robust approach for independent component analysis (ICA) of signals that observations are contaminated with high-level additive noise and/or outliers. The source signals may contain mixtures of both sub-Gaussian and super-Gaussian components, and the number of sources is unknown. Our robust approach includes two procedures. In the first procedure, a robust prewhitening technique is used to reduce the power of additive noise, the dimensionality and the correlation among sources. A cross-validation technique is introduced to estimate the number of sources in this first procedure. In the second procedure, a nonlinear function is derived using the parameterized t-distribution density model. This nonlinear function is robust against the undue influence of outliers fundamentally. Moreover, the stability of the proposed algorithm and the robust property of misestimating the parameters (kurtosis) have been studied. By combining the t-distribution model with a family of light-tailed distributions (sub-Gaussian) model, we can separate the mixture of sub-Gaussian and super-Gaussian source components. Through the analysis of artificially synthesized data and real-world magnetoencephalographic (MEG) data, we illustrate the efficacy of this robust approach.

This paper presents a novel method for decomposing and localizing unaveraged single-trial magnetoencephalographic data based on the independent component analysis (ICA) approach associated with pre- and post-processing techniques. In the pre-processing stage, recorded single-trial raw data are first decomposed into uncorrelated signals with the reduction of high-power additive noise. In the stage of source separation, the decorrelated source signals are further decomposed into independent source components. In the post-processing stage, we perform a source localization procedure to seek a single-dipole map of decomposed individual source components, e.g., evoked responses. The first results of applying the proposed robust ICA approach to single-trial data with phantom and auditory evoked field tasks indicate the following. (1) A source signal is successfully extracted from unaveraged single-trial phantom data. The accuracy of dipole estimation for the decomposed source is even better than that of taking the average of total trials. (2) Not only the behavior and location of individual neuronal sources can be obtained but also the activity strength (amplitude) of evoked responses corresponding to a stimulation trial can be obtained and visualized. Moreover, the dynamics of individual neuronal sources, such as the trial-by-trial variations of the amplitude and location, can be observed. (C) 2002 Elsevier Science B.V. All rights reserved.

An adaptive on-line algorithm extending the learning of learning idea is proposed and theoretically motivated. Relying only on gradient flow information it can be applied to learning continuous functions or distributions, even when no explicit loss function is given and the Hessian is not available. The framework is applied for unsupervised and supervised learning. Its efficiency is demonstrated for drifting and switching non-stationary blind separation tasks of acoustic signals. Furthermore applications to classification (US postal service data set) and time-series prediction in changing environments are presented. (C) 2002 Elsevier Science Ltd. All rights reserved.

We introduce two formulations for training support vector machines, based on considering the L-1 and L-infinity norms instead of the currently used L-2 norm, and maximising the margin between the separating hyperplane and each data sets using L-1 and L-infinity distances. We exploit the geometrical properties of these different norms. and propose what kind of results should be expected for them. Formulations in mathematical programming for linear problems corresponding to L-1 and L-infinity norms are also provided, for both the separable and non-separable cases. We report results obtained for some standard benchmark problems. which confirmed that the performance of all the formulations is similar. As expected, the CPU time required for machines solvable with linear programming is much shorter. (C) 2001 Elsevier Science B.V. All rights reserved.

We construct new algorithms, which use the fourth order cumulant of stochastic variables for the cost function. The multiplicative updating rule here constructed is natural from the homogeneous nature of the Lie group and has numerous merits for the rigorous treatment of the dynamics. As one consequence, the second order convergence is shown. For the cost function, functions invariant under the componentwise scaling are chosen. By identifying points which can be transformed to each other by the scaling, we assume that the dynamics is in a coset space. In our method, a point can move toward any direction in this coset. Thus, no prewhitening is required. (C) 2001 Elsevier Science Ltd. All rights reserved.

In this paper, we introduce a new technique for blind source separation of speech signals. We focus on the temporal structure of the signals. The idea is to apply the decorrelation method proposed by Molgedey and Schuster in the time-frequency domain. Since we are applying separation algorithm on each frequency separately, we have to solve the amplitude and permutation ambiguity properly to reconstruct the separated signals. For solving the amplitude ambiguity, we use the matrix inversion and for the permutation ambiguity, we introduce a method based on the temporal structure of speech signals. We show some results of experiments with both artificially controlled data and speech data recorded in the real environment. © 2001 Elsevier Science B.V. All rights reserved.

Principal component analysis (PCA) and Minor component analysis (MCA) are similar but have different dynamical performances. Unexpectedly, a sequential extraction algorithm for MCA proposed by Luo and Unbehauen [11] does not work for MCA, while it works for PCA. We propose a different sequential-addition algorithm which works for MCA. We also show a conversion mechanism by which any PCA algorithms are converted to dynamically equivalent MCA algorithms and vice versa.

Magnetoencephalography (MEG) is a powerful and non-invasive technique for measuring human brain activity with a high temporal resolution. The motivation for studying MEG data analysis is to extract the essential features from measured data and represent them corresponding to the human brain functions. In this paper a novel MEG data analysis method based on independent component analysis (ICA) approach with pre-processing: and post-processing multistage procedures is proposed. Moreover, several kinds of ICA algorithms are investigated for analyzing MEG; tingle-trial data which is recorded in the experiment of phantom. The analyzed results are presented to illustrate the effectiveness and high performance both in source decomposition by IC:A approaches and source localization by equivalent current dipoles fitting method.

Magnetoencephalography (MEG) is a powerful and non-invasive technique for measuring human brain activity with a high temporal resolution. The motivation for studying MEG data analysis is to extract the essential features from measured data and represent them corresponding to the human brain functions. In this paper a novel MEG data analysis method based on independent component analysis (ICA) approach with pre-processing: and post-processing multistage procedures is proposed. Moreover, several kinds of ICA algorithms are investigated for analyzing MEG; tingle-trial data which is recorded in the experiment of phantom. The analyzed results are presented to illustrate the effectiveness and high performance both in source decomposition by IC:A approaches and source localization by equivalent current dipoles fitting method.

In this paper we deal with the optimisation problem involved in determining the maximal margin separation hyperplane in support vector machines. We consider three different formulations, based on L-2 norm distance (the standard case), L-1 norm, and L-infinity norm. We consider separation in the original space of the data (i.e., there are no kernel transformations).
For any of these cases, we focus on the following problem: having the optimal solution for a given training data set, one is given a new training example. The purpose is to use the information about the solution of the problem without the additional example in order to speed up the new optimisation problem. We also consider the case of reoptimisation after removing an example from the data set.
We report results obtained for some standard benchmark problems.

We study a population decoding paradigm in which the maximum likelihood inference is based on an unfaithful decoding model (UMLI), This is usually the case for neural population decoding because the encoding process of the brain is not exactly known, or because a simplified decoding model is preferred for saving computational cost. We consider an unfaithful decoding model which neglects the pair-wise correlation between neuronal activities, and prove that UMLI is asymptotically efficient when the neuronal correlation is uniform or of limited-range. The performance of UMLI is compared with that of the maximum likelihood inference based on a faithful model and that of the center of mass decoding method. It turns out that UMLI has advantages of decreasing the computational complexity remarkablely and maintaining a high-level decoding accuracy at the same time. The effect of correlation on the decoding accuracy is also discussed.

Learning is a flexible and effective means of extracting the stochastic structure of the environment. It provides an effective method for blind separation and deconvolution in signal processing. Two different types of learning are used, namely batch learning and on-line learning. The batch learning procedure uses all the training examples repeatedly so that its performance is compared to the statistical estimation procedure. On-line learning is more dynamical, updating the current estimate by observing a new datum one by one. On-line learning is slow in general but works well in the changing environment. The present paper gives a unified framework of statistical analysis for batch and on-line learning. The topics include the asymptotic learning curve, generalization error and training error, over-fitting and over-training, efficiency of learning, and an adaptive method of determining learning rate. (C) 1999 Elsevier Science B.V. All rights reserved.

Artificial neural networks (ANNs) are widely used to model low-level neural activities and high-level cognitive functions. In this article, we review the application of statistical inference for learning in ANNs. Statistical inference provides an objective way to derive learning algorithms both for training and for evaluation of the performance of trained ANNs. Solutions to the over-fitting problem by model- selection methods, based on either conventional statistical approaches or on a Bayesian approach, are discussed. The use of supervised and unsupervised learning algorithms for ANNs are reviewed. Training a multilayer ANN by supervised learning is equivalent to nonlinear regression. The ensemble methods, bagging and arching, described here, can be applied to combine ANNs to form a new predictor with improved performance. Unsupervised learning algorithms that are derived either by the Hebbian law for bottom-up self-organization, or by global objective functions for top-down self-organization are also discussed.

A statistical theory for overtraining is proposed. The analysis treats general realizable stochastic neural networks, trained with Kullback-Leibler divergence in the asymptotic case of a large number of training examples. It is shown that the asymptotic gain in the generalization error Is small if we perform early stopping, even if we have access to the optimal stopping time. Considering cross-validation stopping we answer the question: In what ratio the examples should be divided into training and cross-validation sets in order to obtain the optimum performance. Although cross-validated early stopping is useless in the asymptotic region, it surely decreases the generalization error in the nonasymptotic region. Our large scale simulations done on a CM5 are in nice agreement with our analytical findings.

An adaptive on-line algorithm extending the learning of learning idea is proposed and theoretically motivated. Relying only on gradient flow information it can be applied, to learning continuous functions or distributions, even when no explicit loss function is given and the Hessian is not available. Its efficiency is demonstrated for a non-stationary blind separation task of acoustic signals.

In order to avoid overfitting in neural learning, a regularization term is added to the loss function to be minimized. It is naturally derived from the Bayesian standpoint. The present paper studies how to determine the regularization constant from the points of view of the empirical Bayes approach, the maximum description length (MDL) approach, and the network information criterion (NIC) approach. The asymptotic statistical analysis is given to elucidate their differences. These approaches are tightly connected with the method of model selection. The superiority of the NIC is shown from this analysis.

The universal asymptotic scaling laws proposed by Amari et al. are studied in large scale simulations using a CM5. Small stochastic multilayer feedforward networks trained with backpropagation are investigated. In the range of a large number of training patterns t, the asymptotic generalization error scales as 1/t as predicted. For a medium range t a faster 1/t(2) scaling is observed. This effect is explained by using higher order corrections of the likelihood expansion. It is shown for small t that the scaling law changes drastically, when the network undergoes a transition from strong overfitting to effective learning.

Neural networks have attracted attention due to their capability to perform nonlinear function approximation. In this paper, in order to better understand this capability, a new theorem on an integral transform was derived by applying ridge functions to neural networks. From the theorem, it is possible to obtain approximation bounds which clarify the quantitative relationship between the function approximation accuracy and the number of nodes in the hidden layer. The theorem indicates that the approximation accuracy depends on the smoothness of the target function. Furthermore, the theorem also shows that this type of approximation method differs from usual methods and is able to escape the so-called ''curse of dimensionality,'' in which the approximation accuracy depends strongly of the input dimension of the function and deteriorates exponentially.

Neural networks are widely known to provide a method of approximating nonlinear functions. In order to clarify its approximation ability, a new theorem on an integral transform of ridge functions is presented. By using this theorem, an approximation bound, which evaluates the quantitative relationship between the approximation accuracy and the number of elements in the hidden layer, can be obtained. This result shows that the approximation accuracy depends on the smoothness of target functions. It also shows that the approximation methods which use ridge functions are free from the 'curse of dimensionality'.

The problem of model selection, or determination of the number of hidden units, can be approached statistically, by generalizing Akaike's information Criterion (AIC) to be applicable to unfaithful (i.e., unrealizable) models with general loss criteria including regularization terms. The relation between the training error and the generalization error is studied in terms of the number of the training examples and the complexity of It network which reduces to the number of parameters in the ordinary statistical theory of the AIC. This relation leads to a new Network Information Criterion (NIC) which is useful for selecting the optimal network model based on a given training set.

The present paper elucidates a universal property of learning curves, which shows how the generalization error, training error, and the complexity of the underlying stochastic machine are related and how the behavior of a stochastic machine is improved as the number of training examples increases. The error is measured by the entropic loss. It is proved that the generalization error converges to H0, the entropy of the conditional distribution of the true machine, as H0 + m*/(2t), while the training error converges as H0 - m*/(2t), where t is the number of examples and m* shows the complexity of the network. When the model is faithful, implying that the true machine is in the model, m* is reduced to m, the number of modifiable parameters. This is a universal law because it holds for any regular machine irrespective of its structure under the maximum likelihood estimator. Similar relations are obtained for the Bayes and Gibbs learning algorithms. These learning curves show the relation among the accuracy of learning, the complexity of a model, and the number of training examples.