Background
Database Migration 是一個非常龐大的工程, 但因為自己的經驗不足所以沒辦法做到 zero downtime.
在這個工程中, 有幾個需要處理的 issue
Oracle 的資料型別如何轉移對應？比如說 Oracle 的 timestamp 要對應 Postgres 的什麼型別資料？
Oracle 的資料庫物件如何轉移對應？ Sequence, Procedures function, View, etc. 要怎麼對應？
轉移時, 線上服務要怎麼繼續運行？
轉移後, 要怎麼測試確保 DB 資料一切正常, ORM (Mybatis 裡面寫了很多 Oracle function) 工具可以正常運作？
Tools
在這過程中, 我花了大約兩天研究了一些工具.
Ora2Pg - 算是網路上資源最完整的工具, 但說不上好用, 使用 Perl 開發的工具, 速度也不算快.
AWS Schema Conversion Tool - 只支援 DDL 的轉換, Data 要自己想辦法處理資料 ETL (除非你資料都放在 AWS 上面, 而公司的狀態, 不太可能為了這件事 dump 資料上 AWS XD.
Oraface - 針對 Oracle 的 function 寫了很多轉換, 要跑在 C 語言環境, 也是有點麻煩.
fullconvert - 試用版有很多限制, 而正式版要錢之外, 也不太確定能不能完整跑完全不資料 XD.
dbconvert - 只支援 win 環境
convert-in - 只支援 win 環境
找過 Taipei 的幾間廠商詢問過, 但因為公司資料量太小非常不划算 XD.
Ora2Pg
整個測試跑下來似乎只有 Ora2Pg 能跑個 80%, 一些 Oracle functions 要想辦法處理.
整個架構很簡單,
為了減少網路傳輸, 我在 Postgres 環境下建立了 Converter 的環境.
Ora2Pg Setup
- 需要 perl5 以上的環境.
# check your perl version,
# if it does't work, you should install perl
perl -v
# setup perl by yum, it's will setup all perl packagies
yum install perl*
# and setup cpan
yum install cpan
# then check perl
perl -v
# this is my perl info
This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi
(with 39 registered patches, see perl -V for more detail)
Copyright 1987-2012, Larry Wall
Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.
Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl". If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.
其他安裝步驟跟官方差不多
# make and install
cd ora2pg-20.0
perl Makefile.PL
make && make install
# set env
echo "PERL5LIB=$PERL5LIB:/root/perl5/lib/perl5" >> ~/.bash_profile
echo "ORACLE_HOME=$ORACLE_HOME:/u01/app/oracle/product/11.2.0/EE" >> ~/.bash_profile
echo "LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib" >> ~/.bash_profile
source ~/.bash_profile
# install DBD:Oracle
perl -MCPAN -e 'install DBD::Oracle'
# make and install DBD:Oracle (Oracle connetion driver)
cd ~/.cpan/build/DBD-Oracle*
perl Makefile.PL
make
make install
寫個 script 檢查 peril 運行.
#!/usr/bin/perl
use ExtUtils::Installed;
my $inst = ExtUtils::Installed->new();
print join "\n",$inst->modules();
Check perl 相關依賴套件都 Ready.
# 執行看看
chmod +x check.pl
perl check.pl
# 順利的話, stdout 會輸出
DBD::Oracle
Ora2Pg
Perl
Test::NoWarnings
建立 Peril 專案
這專案看結構命名, 可以很清楚的知道 Data Object 的處理程式位置.
ora2pg --project_base ~/ora2pg_projects/ --init_project project_name
Creating project test_project.
/app/migration/test_project/
schema/
dblinks/
directories/
functions/
grants/
mviews/
packages/
partitions/
procedures/
sequences/
synonyms/
tables/
tablespaces/
triggers/
types/
views/
sources/
functions/
mviews/
packages/
partitions/
procedures/
triggers/
types/
views/
data/
config/
reports/
Generating generic configuration file
Creating script export_schema.sh to automate all exports.
Creating script import_all.sh to automate all imports.
Ora2Pg Configuration Setting
在專案目錄底下的 config/ora2pg.conf 把 Oracle 的連線資訊設定一下.
# Oracle 作為 source, 設定好相關 oracle driver 連線方式.
ORACLE_DSN dbi:Oracle:host=localhost;sid=EE;port=1521
ORACLE_USER system
ORACLE_PWD foobar
Migrate Data
一樣在專案資料目錄底下, 會有兩個檔案.
export_schema.sh用來匯出 data objects schema.import_all.sh用來匯入 data objects schema.
接下來就是重複這兩個指令, 直到所有資料問題都排除.
# execute
./export_schema.sh
# stdout
[========================>] 52/52 tables (100.0%) end of scanning.
[========================>] 11/11 objects types (100.0%) end of objects auditing.
Running: ora2pg -p -t TABLE -o table.sql -b ./schema/tables -c ./config/ora2pg.conf
[========================>] 52/52 tables (100.0%) end of scanning.
[========================>] 52/52 tables (100.0%) end of table export.
Running: ora2pg -p -t PACKAGE -o package.sql -b ./schema/packages -c ./config/ora2pg.conf
[========================>] 1/1 packages (100.0%) end of output.
Running: ora2pg -p -t VIEW -o view.sql -b ./schema/views -c ./config/ora2pg.conf
[========================>] 0/0 views (100.0%) end of output.
Running: ora2pg -p -t GRANT -o grant.sql -b ./schema/grants -c ./config/ora2pg.conf
WARNING: Exporting privilege as non DBA user is not allowed, see USER_GRANT
Running: ora2pg -p -t SEQUENCE -o sequence.sql -b ./schema/sequences -c ./config/ora2pg.conf
[========================>] 1/1 sequences (100.0%) end of output.
Running: ora2pg -p -t TRIGGER -o trigger.sql -b ./schema/triggers -c ./config/ora2pg.conf
[========================>] 82/82 triggers (100.0%) end of output.
Running: ora2pg -p -t FUNCTION -o function.sql -b ./schema/functions -c ./config/ora2pg.conf
[========================>] 4/4 functions (100.0%) end of functions export.
Running: ora2pg -p -t PROCEDURE -o procedure.sql -b ./schema/procedures -c ./config/ora2pg.conf
[========================>] 0/0 procedures (100.0%) end of procedures export.
Running: ora2pg -p -t TABLESPACE -o tablespace.sql -b ./schema/tablespaces -c ./config/ora2pg.conf
WARNING: Exporting tablespace as non DBA user is not allowed, see USER_GRANT
Running: ora2pg -p -t PARTITION -o partition.sql -b ./schema/partitions -c ./config/ora2pg.conf
[========================>] 0/0 partitions (100.0%) end of output.
Running: ora2pg -p -t TYPE -o type.sql -b ./schema/types -c ./config/ora2pg.conf
[========================>] 0/0 types (100.0%) end of output.
Running: ora2pg -p -t MVIEW -o mview.sql -b ./schema/mviews -c ./config/ora2pg.conf
[========================>] 0/0 materialized views (100.0%) end of output.
Running: ora2pg -p -t DBLINK -o dblink.sql -b ./schema/dblinks -c ./config/ora2pg.conf
[========================>] 0/0 dblink (100.0%) end of output.
Running: ora2pg -p -t SYNONYM -o synonym.sql -b ./schema/synonyms -c ./config/ora2pg.conf
[========================>] 0/0 synonyms (100.0%) end of output.
Running: ora2pg -p -t DIRECTORY -o directorie.sql -b ./schema/directories -c ./config/ora2pg.conf
[========================>] 0/0 directory (100.0%) end of output.
Running: ora2pg -t PACKAGE -o package.sql -b ./sources/packages -c ./config/ora2pg.conf
[========================>] 1/1 packages (100.0%) end of output.
Running: ora2pg -t VIEW -o view.sql -b ./sources/views -c ./config/ora2pg.conf
[========================>] 0/0 views (100.0%) end of output.
Running: ora2pg -t TRIGGER -o trigger.sql -b ./sources/triggers -c ./config/ora2pg.conf
[========================>] 82/82 triggers (100.0%) end of output.
Running: ora2pg -t FUNCTION -o function.sql -b ./sources/functions -c ./config/ora2pg.conf
[========================>] 4/4 functions (100.0%) end of functions export.
Running: ora2pg -t PROCEDURE -o procedure.sql -b ./sources/procedures -c ./config/ora2pg.conf
[========================>] 0/0 procedures (100.0%) end of procedures export.
Running: ora2pg -t PARTITION -o partition.sql -b ./sources/partitions -c ./config/ora2pg.conf
[========================>] 0/0 partitions (100.0%) end of output.
Running: ora2pg -t TYPE -o type.sql -b ./sources/types -c ./config/ora2pg.conf
[========================>] 0/0 types (100.0%) end of output.
Running: ora2pg -t MVIEW -o mview.sql -b ./sources/mviews -c ./config/ora2pg.conf
[========================>] 0/0 materialized views (100.0%) end of output.
To extract data use the following command:
# you can follow this command export data
ora2pg -t COPY -o data.sql -b ./data -c ./config/ora2pg.conf
在 Migrate 執行之前
在進行 Migrate 工程之前, 我最先思考的是如何測試 Migrate 成功？ 所以我做了一些測試與腳本.
驗證資料筆數.
驗證 view 與 Procedures 的數量, 並且對其使用情境寫了基本 pass testings.
驗證應用程式內的 ORM (MyBatis) 都能正常執行, 要求團隊協力寫了所有 ORM 的 pass testings.
整個工程大約是花了 2 個月建立測試計畫, 1 天做 Migrate 資料轉移.
因為移植的資料庫是公司內部的 ERP 並沒有太大下線壓力, 公司容許一天下線 (其實就是技術部門週末加班) 而已, 因為有完整的測試計畫, 移植後的 bug 修正也都在一個禮拜內修正完畢 (4 人團隊).