メニュー

技術ブログ

Denso IT Lab.社員による技術情報の紹介

Denso IT Laboratory researcher's blog sites

数式

RSS

ページトップへ

「変分ベイズ学習」の学習

変分ベイズ学習 (機械学習プロフェッショナルシリーズ)

機械学習プロフェッショナルシリーズの変分ベイズ学習を読んでいる時に,もっとも引っかかった変分ベイズの停留条件の導出の式展開を真面目にやってみました.
汎関数微分や試験関数の厳密性は勘弁してください・・・・・・.
間違っているところがあれば,指摘いただけるとうれしいです.

自由エネルギー$F$が停留する条件を考えると,それがベイズ事後分布の条件になるよ・・・というところです.

##

自由エネルギー$F$を$r_s$で微分し,汎関数微分の値が常に0となる停留条件を考える.
$r_s$の関数として,自由エネルギー$F$を整理する.

$\scriptsize{ \begin{eqnarray}
F({r_s}) &=& \int {\prod\limits_{m = 1}^S {{r_m}({\omega _m}) \cdot \log \frac{{\prod\limits_{m = 1}^S {{r_m}({\omega _m})} }}{{p(D|\omega ) \cdot \prod\limits_{m = 1}^S {p({\omega _m})} }}} } d\omega \tag{1} \\
&=& \int {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot {r_s}({\omega _s})} \cdot \log \frac{{\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot {r_s}({\omega _s})} }}{{p(D|\omega ) \cdot \prod\limits_{m \ne s} {p({\omega _m})} \cdot p({\omega _s})}}} d\omega \tag{2} \\
&=& \iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot {r_s}({\omega _s})} \cdot \log \frac{{\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot {r_s}({\omega _s})} }}{{p(D|\omega ) \cdot \prod\limits_{m \ne s} {p({\omega _m})} \cdot p({\omega _s})}}}d{\omega _{\backslash s}}d{\omega _s} \tag{3} \\
\end{eqnarray} } $

さらに,確率密度関数の全積分の値が$1$である制約条件をラグランジュの未定乗数をかけて,この自由エネルギーに追加する.

$\scriptsize{ \begin{eqnarray}
1 &=& \int {{r_s}({\omega _s})d{\omega _s}} \tag{4} \\
F({r_s}) &=& \iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot {r_s}({\omega _s})} \cdot \log \frac{{\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot {r_s}({\omega _s})} }}{{p(D|\omega ) \cdot \prod\limits_{m \ne s} {p({\omega _m})} \cdot p({\omega _s})}}}d{\omega _{\backslash s}}d{\omega _s} \\ & & + \lambda \cdot (1 – \int {{r_s}({\omega _s})d{\omega _s}} ) \tag{5} \\
\end{eqnarray} } $

汎関数微分の定義微分を考える.

$\scriptsize{ \begin{eqnarray}
\delta F &=& \int {\frac{{\delta F[\varphi ({\omega _s})]}}{{\delta \varphi ({\omega _s}^\prime )}}} \cdot f({\omega _s}^\prime )d{\omega _s}^\prime \tag{6} \\
&=& \mathop {\lim }\limits_{\varepsilon \to 0} \frac{{F(\varphi ({\omega _s}) + \varepsilon \cdot f({\omega _s})) – F(\varphi ({\omega _s}))}}{\varepsilon } \tag{7} \\
\end{eqnarray} } $

試験関数と汎関数に代入する関数を以下のように定義する.

$\scriptsize{ \begin{eqnarray}
f({\omega _s}) & \triangleq & \Delta {r_s}({\omega _s}) \tag{8} \\
\varphi ({\omega _s}) & \triangleq & {r_s}({\omega _s}) \tag{9} \\
\end{eqnarray} } $

この定義微分$(6)(7)$に,定義$(8)(9)$を代入する.

$\scriptsize{ \begin{eqnarray}
\delta F &=& \int {\frac{{\delta F[{r_s}({\omega _s})]}}{{\delta {r_s}({\omega _s}^\prime )}}} \cdot \Delta {r_s}({\omega _s}^\prime )d{\omega _s}^\prime \tag{10} \\
\delta F &=& \mathop {\lim }\limits_{\varepsilon \to 0} \frac{{F({r_s}({\omega _s}) + \varepsilon \cdot \Delta {r_s}({\omega _s})) – F({r_s}({\omega _s}))}}{\varepsilon } \tag{11} \\
\end{eqnarray} } $

式$(11)$に,自由エネルギー$(5)$を代入する.

$\scriptsize{ \begin{eqnarray}
\delta F &=& \mathop {\lim }\limits_{\varepsilon \to 0} \frac{1}{\varepsilon }\iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot ({r_s}({\omega _s}) + \varepsilon \cdot \Delta {r_s}({\omega _s}))} \cdot \log \frac{{\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot ({r_s}({\omega _s}) + \varepsilon \cdot \Delta {r_s}({\omega _s}))} }}{{p(D|\omega ) \cdot \prod\limits_{m \ne s} {p({\omega _m})} \cdot p({\omega _s})}}}d{\omega _{\backslash s}}d{\omega _s} \\
& & – \frac{1}{\varepsilon }\iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot {r_s}({\omega _s})} \cdot \log \frac{{\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot {r_s}({\omega _s})} }}{{p(D|\omega ) \cdot \prod\limits_{m \ne s} {p({\omega _m})} \cdot p({\omega _s})}}}d{\omega _{\backslash s}}d{\omega _s} \\
& & + \frac{1}{\varepsilon }\lambda \cdot (1 – \int {{r_s}({\omega _s}) + \varepsilon \cdot \Delta {r_s}({\omega _s})d{\omega _s}} ) – \frac{1}{\varepsilon }\lambda \cdot (1 – \int {{r_s}({\omega _s})d{\omega _s}} ) \\
&=& \mathop {\lim }\limits_{\varepsilon \to 0} \frac{1}{\varepsilon }\iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot ({r_s}({\omega _s}) + \varepsilon \cdot \Delta {r_s}({\omega _s}))} \cdot \left[ {\log ({r_s}({\omega _s}) + \varepsilon \cdot \Delta {r_s}({\omega _s})) + \log \frac{{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }}{{p(D|\omega ) \cdot \prod\limits_{m \ne s} {p({\omega _m})} \cdot p({\omega _s})}}} \right]}d{\omega _{\backslash s}}d{\omega _s} \\
& & – \frac{1}{\varepsilon }\iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot {r_s}({\omega _s})} \cdot \log \frac{{\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot {r_s}({\omega _s})} }}{{p(D|\omega ) \cdot \prod\limits_{m \ne s} {p({\omega _m})} \cdot p({\omega _s})}}}d{\omega _{\backslash s}}d{\omega _s} \\
& & – \frac{1}{\varepsilon }\lambda \cdot \int {\varepsilon \cdot \Delta {r_s}({\omega _s})d{\omega _s}}
\end{eqnarray} } $

${\log ({r_s}({\omega _s}) + \varepsilon \cdot \Delta {r_s}({\omega _s}))}$を線形近似する.

$\scriptsize{
\begin{eqnarray}
\frac{{\delta F}}{{\delta {r_s}}} &=& \mathop {\lim }\limits_{\varepsilon \to 0} \frac{1}{\varepsilon }\iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot \left[ {{r_s}({\omega _s}) + \varepsilon \cdot \Delta {r_s}({\omega _s})} \right]} \cdot \left[ {\log {r_s}({\omega _s}) + \frac{{\varepsilon \cdot \Delta {r_s}({\omega _s})}}{{{r_s}({\omega _s})}} + \log \frac{{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }}{{p(D|\omega ) \cdot \prod\limits_{m \ne s} {p({\omega _m})} \cdot p({\omega _s})}}} \right]}d{\omega _{\backslash s}}d{\omega _s} \\
& & – \frac{1}{\varepsilon }\iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot {r_s}({\omega _s})} \cdot \left[ {\log {r_s}({\omega _s}) + \log \frac{{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }}{{p(D|\omega ) \cdot \prod\limits_{m \ne s} {p({\omega _m})} \cdot p({\omega _s})}}} \right]}d{\omega _{\backslash s}}d{\omega _s} \\
& & – \lambda \cdot \int {\Delta {r_s}({\omega _s})d{\omega _s}} \\
&=& \mathop {\lim }\limits_{\varepsilon \to 0} \frac{1}{\varepsilon }\iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot {r_s}({\omega _s})} \cdot \frac{{\varepsilon \cdot \Delta {r_s}({\omega _s})}}{{{r_s}({\omega _s})}}}d{\omega _{\backslash s}}d{\omega _s} \\
& & + \frac{1}{\varepsilon }\iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot \varepsilon \cdot \Delta {r_s}({\omega _s})} \cdot \left[ {\log {r_s}({\omega _s}) + \frac{{\varepsilon \cdot \Delta {r_s}({\omega _s})}}{{{r_s}({\omega _s})}} + \log \frac{{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }}{{p(D|\omega ) \cdot \prod\limits_{m \ne s} {p({\omega _m})} \cdot p({\omega _s})}}} \right]}d{\omega _{\backslash s}}d{\omega _s} \\
& & – \lambda \cdot \int {\Delta {r_s}({\omega _s})d{\omega _s}} \\
&=& \mathop {\lim }\limits_{\varepsilon \to 0} \iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot \Delta {r_s}({\omega _s})} }d{\omega _{\backslash s}}d{\omega _s} \\
& & + \iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot \Delta {r_s}({\omega _s})} \cdot \left[ {\log {r_s}({\omega _s}) + \frac{{\varepsilon \cdot \Delta {r_s}({\omega _s})}}{{{r_s}({\omega _s})}} + \log \frac{{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }}{{p(D|\omega ) \cdot \prod\limits_{m \ne s} {p({\omega _m})} \cdot p({\omega _s})}}} \right]}d{\omega _{\backslash s}}d{\omega _s} \\
& & – \lambda \cdot \int {\Delta {r_s}({\omega _s})d{\omega _s}} \\
&=& \iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot \Delta {r_s}({\omega _s})} }d{\omega _{\backslash s}}d{\omega _s} \\
& & + \iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot \Delta {r_s}({\omega _s})} \cdot \left[ {\log {r_s}({\omega _s}) + \log \frac{{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }}{{p(D|\omega ) \cdot \prod\limits_{m \ne s} {p({\omega _m})} }} – \log p({\omega _s})} \right]}d{\omega _{\backslash s}}d{\omega _s} \\
& & – \lambda \cdot \int {\Delta {r_s}({\omega _s})d{\omega _s}} \\
&=& \iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot \Delta {r_s}({\omega _s})} }d{\omega _{\backslash s}}d{\omega _s} \\
& & + \iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot \Delta {r_s}({\omega _s})} \cdot \log \frac{{{r_s}({\omega _s})}}{{p({\omega _s})}}}d{\omega _{\backslash s}}d{\omega _s} \\
& & + \iint {\prod\limits_{m \ne s} {{r_m}({\omega _m}) \cdot \Delta {r_s}({\omega _s})} \cdot \log \frac{{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }}{{p(D|\omega ) \cdot \prod\limits_{m \ne s} {p({\omega _m})} }}}d{\omega _{\backslash s}}d{\omega _s} \\
& & – \lambda \cdot \int {\Delta {r_s}({\omega _s})d{\omega _s}} \\
&=& \int {\left[ {\int {\prod\limits_{m \ne s} {{r_m}({\omega _m})} d{\omega _{\backslash s}}} } \right]} \Delta {r_s}({\omega _s})d{\omega _s} \\
& & + \int {\left[ {\int {\prod\limits_{m \ne s} {{r_m}({\omega _m})} d{\omega _{\backslash s}}} } \right]} \cdot \log \frac{{{r_s}({\omega _s})}}{{p({\omega _s})}}\Delta {r_s}({\omega _s})d{\omega _s} \\
& & + \int {\left[ {\int {\log \frac{1}{{p(D|\omega )}} \cdot \prod\limits_{m \ne s} {{r_m}({\omega _m})} d{\omega _{\backslash s}}} } \right]} \cdot \Delta {r_s}({\omega _s})d{\omega _s}_s \\
& & + \int {\left[ {\int {\log \frac{{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }}{{\prod\limits_{m \ne s} {p({\omega _m})} }} \cdot \prod\limits_{m \ne s} {{r_m}({\omega _m})} d{\omega _{\backslash s}}} } \right]} \cdot \Delta {r_s}({\omega _s})d{\omega _s} \\
& & – \lambda \cdot \int {\Delta {r_s}({\omega _s})d{\omega _s}} \\
&=& \int {\Delta {r_s}({\omega _s})d{\omega _s}} \\
& & + \int {\log \frac{{{r_s}({\omega _s})}}{{p({\omega _s})}}\Delta {r_s}({\omega _s})d{\omega _s}} \\
& & + \int {\left[ {\int {\log \frac{1}{{p(D|\omega )}} \cdot \prod\limits_{m \ne s} {{r_m}({\omega _m})} d{\omega _{\backslash s}}} } \right]} \cdot \Delta {r_s}({\omega _s})d{\omega _s}_s \\
& & + \int {const.} \cdot \Delta {r_s}({\omega _s})d{\omega _s} \\
& & – \lambda \cdot \int {\Delta {r_s}({\omega _s})d{\omega _s}} \\
\end{eqnarray}
}$

${r_s}({\omega _s})$に無関係な項は定数としてまとめる.また,${{\omega _{\backslash s}}}$の期待値計算としてまとめられる項は,期待値の表記でまとめる.

$\scriptsize{ \begin{eqnarray}
\frac{{\delta F}}{{\delta {r_s}}} &=& \int {\left[ {\log \frac{{{r_s}({\omega _s})}}{{p({\omega _s})}} + {{\left\langle {\log \frac{1}{{p(D|\omega )}}} \right\rangle }_{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }} + 1 – \lambda + const.} \right] \cdot \Delta {r_s}({\omega _s})} d{\omega _s} \tag{12} \\
\end{eqnarray} } $

式$(12)$のラグランジュの未定乗数に定数を押し付ける.

$\scriptsize{ \begin{eqnarray}
\frac{{\delta F}}{{\delta {r_s}}} &=& \int {\left[ {\log \frac{{{r_s}({\omega _s})}}{{p({\omega _s})}} + {{\left\langle {\log \frac{1}{{p(D|\omega )}}} \right\rangle }_{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }} + \lambda’ } \right] \cdot \Delta {r_s}({\omega _s})} d{\omega _s} \tag{13}
\end{eqnarray} } $

となり,${\log \frac{{{r_s}({\omega _s})}}{{p({\omega _s})}} + {{\left\langle {\log \frac{1}{{p(D|\omega )}}} \right\rangle }_{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }} + \lambda’}$が0となるときに,汎関数微分の値は常に0となる.この条件を展開し,停留条件を得ることができる.

$\scriptsize{ \begin{eqnarray}
0 &=& \log \frac{{{r_s}({\omega _s})}}{{p({\omega _s})}} + {\left\langle {\log \frac{1}{{p(D|\omega )}}} \right\rangle _{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }} + \lambda’ \tag{14} \\
– \lambda’ &=& \log \frac{{{r_s}({\omega _s})}}{{p({\omega _s})}} + {\left\langle {\log \frac{1}{{p(D|\omega )}}} \right\rangle _{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }} \tag{15} \\
\exp ( – \lambda’ ) &=& \exp (\log \frac{{{r_s}({\omega _s})}}{{p({\omega _s})}}) \cdot \exp {\left\langle {\log \frac{1}{{p(D|\omega )}}} \right\rangle _{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }} \tag{16} \\
\exp ( – \lambda’ ) &=& \frac{{{r_s}({\omega _s})}}{{p({\omega _s})}} \cdot \exp {\left\langle {\log \frac{1}{{p(D|\omega )}}} \right\rangle _{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }} \tag{17} \\
{r_s}({\omega _s}) &=& \exp ( – \lambda’ ) \cdot p({\omega _s}) \cdot \exp {\left\langle {\log p(D|\omega )} \right\rangle _{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }} \tag{18} \\
{r_s}({\omega _s}) & \propto& p({\omega _s}) \cdot \exp {\left\langle {\log p(D|\omega )} \right\rangle _{\prod\limits_{m \ne s} {{r_m}({\omega _m})} }} \tag{19} \\
\end{eqnarray} } $

このエントリーをはてなブックマークに追加

MIRU2016で会いましょう

MIRU2016では,弊社の安倍,近藤,佐藤,吉田の四人が発表します.

安倍・・・”整数基底分解によるDeep Neural Networkの時間・空間計算量削減”

近藤・・・”組み込み向けDNNライブラリの実装と演算性能評価”
     デモアプリケーション Demitasse – Image Recognition Cam

佐藤・・・”APAC: Augmented PAttern Classification with Neural Networks”

吉田・・・”二重特徴表現を使った特定物体認識”

研究や技術のことはもちろん,デンソーアイティーラボラトリに興味のある方は,ぜひ,各社員にお声掛けください!

demitasse

このエントリーをはてなブックマークに追加

How to build matcaffe on OSX10.11.5 and MATLAB2016a

I guess that many engineers have some troubles when they try to build matcaffe on OSX.

1. Link error – google::protobuf
2. Link error – mex functions – not found _mxArrayToString
3. Link error – dyld: Library not loaded: @rpath/libmkl_intel_lp64.dylib

I found a way to avoid these troubles, so I will write down about it.

At first, my system is

– Mac Pro (Late 2013)
– OSX 10.11.5
– MATLAB 2016a
– Intel Math Kernel Library(or OpenBLAS via homebrew)
– without Python interface
– without CUDA

1. Install some packages according to “OSX Installation”.

http://caffe.berkeleyvision.org/install_osx.html

2. Install Intel Math Kernel Library.

https://software.intel.com/en-us/intel-mkl/try-buy

Or, you can use OpenBLAS or Accelerate.framework.

3. Edit Makefile.config.

You have to edit Makefile.config very carefully.

Make it use only CPU.

# CPU-only switch (uncomment to build without GPU support).
CPU_ONLY := 1

Specify compiler.

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
CUSTOM_CXX := /usr/bin/clang++

Set BLAS path according to your library.
This sample is for Intel Math Kernel Library.

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := mkl
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
BLAS_INCLUDE := /opt/intel/compilers_and_libraries_2016.3.170/mac/mkl/include
BLAS_LIB := /opt/intel/compilers_and_libraries_2016.3.170/mac/mkl/lib

Set MATLAB path to build matcaffe.

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
MATLAB_DIR := /Applications/MATLAB_R2016a.app

This is the most important point.
Remove paths for python and keep the under order of LIBRARY_DIRS’s elements in order to avoid link error.

# Whatever else you find you need goes here.
INCLUDE_DIRS := /usr/local/include
LIBRARY_DIRS := /usr/local/lib $(MATLAB_DIR)/bin/maci64 /usr/lib

Next, build caffe.

4. Build caffe and matcaffe

If you want to use Intel Math Kernel Library, you have to execute this command before it.
I recommend you should copy this command into .bash_profile.

# build caffe
source /opt/intel/compilers_and_libraries/mac/mkl/bin/mklvars.sh intel64 ilp64

cd caffe
make all
make all test
make runtest
make all matcaffe

If failed to build caffe, check Makefile.config again.

5. How to use matcaffe.

You have to run MATLAB from terminal.app because MATLAB does not load environment variables when it runs via Finder.app……

cd caffe
# You have set this environment variable
# in order to avoid caffe load MATLAB's tiff library....
export DYLD_INSERT_LIBRARIES=/usr/local/opt/libtiff/lib/libtiff.5.dylib
/Applications/MATLAB_R2016a.app/bin/matlab &

You can use matcaffe in MATLAB console.

cd caffe
addpath('./matlab');
caffe.set_mode_cpu();

That’s all! Have fun!

このエントリーをはてなブックマークに追加