Postgresql on ZRJ | 学习笔记

postgresql 迁移数据到 MySQL

Wed, 31 Jul 2019 20:52:30 +0800

最近在搞 tidb，想找个方法往 MySQL 协议中快速写入数据，在 postgresql 下有 copy，找了下，MySQL 下也有 load data，其本质也是一个 insert，看到这里 https://blog.csdn.net/seven_3306/article/details/9237495 有一个思路是不走落地文件，挖坑待填

===========

2019-9-4 回来填坑了，具体的方法如下

public static InputStream getPostgresqlInputStream(final String ds) throws SQLException,
        ClassNotFoundException, IOException {
    final PipedOutputStream os = new PipedOutputStream();
    Class.forName("org.postgresql.Driver");
    (new Thread() {
        @Override
        public void run() {
            Connection connection = null;
            try {
                connection = DriverManager.getConnection(
                        "jdbc:postgresql://...",
                        "...", "...");
                CopyManager cm = new CopyManager(
                        (BaseConnection) connection);                
                String COPY_CMD = "COPY (select ... from ... where ds between "+ds+"01 and "+ds+"31 ) to STDOUT";
                @SuppressWarnings("unused")
                long affectedRowCount = cm.copyOut(COPY_CMD, os);
            } catch (Exception e) {
                e.printStackTrace();
            } finally {
                if (connection != null) {
                    try {
                        connection.close();
                    } catch (SQLException e) {
                        e.printStackTrace();
                    }
                }
            }
            try {
                os.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }).start();
    PipedInputStream in = new PipedInputStream();
    in.connect(os);
    return in;
}

public static void threadTransfer(boolean fromTpgOrFromFile, String ds, String csvFile) throws SQLException, ClassNotFoundException, IOException {
    Class.forName("com.mysql.jdbc.Driver");
    String localConnectionUrl = "jdbc:mysql://...";
    
    Random randomizer = new Random();
    java.util.List<String[]> random_pool = new ArrayList<String[]>();
    random_pool.add(new String[]{"..."});
    String[] randomRow1 = random_pool.get(randomizer.nextInt(random_pool.size()));
    System.out.println(randomRow1[0]);
    
    String tidbConnectionUrl = "jdbc:mysql://" + randomRow1[0] + ":4000/...?useSSL=false&characterEncoding=UTF-8";
    Connection conn = DriverManager.getConnection(tidbConnectionUrl, "root", "");
    conn.prepareStatement("set @@tidb_batch_delete = ON;").execute();
    conn.prepareStatement("set @@tidb_batch_insert = ON;").execute();
    if (fromTpgOrFromFile) {
        String loadDataSql = "LOAD DATA LOCAL INFILE 'sql.csv' IGNORE INTO TABLE ...";
        PreparedStatement statement = (PreparedStatement) conn
                .prepareStatement(loadDataSql);
        statement.setLocalInfileInputStream(getPostgresqlInputStream(ds));
        int result = statement.executeUpdate();
        System.out.println(result);
    } else {
        String loadDataSql = "LOAD DATA LOCAL INFILE 'sql.csv' IGNORE INTO TABLE ... FIELDS ENCLOSED BY '\"' TERMINATED BY ',' LINES TERMINATED BY '\n' IGNORE 1 LINES ;";
        PreparedStatement statement = (PreparedStatement) conn
                .prepareStatement(loadDataSql);
        statement.setLocalInfileInputStream(new FileInputStream("/data/my_database/...." + csvFile + ".csv"));
        int result = statement.executeUpdate();
        System.out.println(result);
    }
    System.out.println(System.currentTimeMillis());
}

postgresql 文本索引 spgist 与 btree 区别

Tue, 06 Nov 2018 22:17:31 +0800

postgresql 对文本的索引有 spgist 和 btree 两种，同样一张表，行数 140w 行左右，建立 btree 索引，执行一次 like starts with，冷启动大约 1.4s，反复多跑一次，稳定在 1.1s

改成 spgist 索引呢，瞬间降到 0.03s，到底是什么黑科技，能这么明显的提速

搜了一下，看到这里 http://www.postgres.cn/docs/9.6/spgist.html 大概是类似前缀索引和查找的套路

postgresql 死锁检查与处理

Sun, 30 Sep 2018 10:16:11 +0800

今天在 postgresql 上执行一个 DDL 的时候卡死，其实就是一个 alter table add column 的语句

那么，首先可以用这个 SQL 来看有哪些执行中的查询

select query, * from pg_stat_activity where query != '' order by query_start asc;

但是从这里有两个问题，一是只能看到自己名下的，别人的 SQL 看不到具体 query，二是通过这个结果看不出到底是啥 SQL 阻塞了自己的 DDL

Google 一下，看到这里， https://stackoverflow.com/questions/26489244/how-to-detect-query-which-holds-the-lock-in-postgres

select pid, 
       usename, 
       pg_blocking_pids(pid) as blocked_by, 
       query as blocked_query
from pg_stat_activity
where cardinality(pg_blocking_pids(pid)) > 0;

但是可惜，这种方法只能支持到 postgresql 9.6 以上的，而我们是 9.5，但是没关系，下面又给出了另外一种方式

SELECT
  COALESCE(blockingl.relation::regclass::text,blockingl.locktype) as locked_item,
  now() - blockeda.query_start AS waiting_duration, blockeda.pid AS blocked_pid,
  blockeda.query as blocked_query, blockedl.mode as blocked_mode,
  blockinga.pid AS blocking_pid, blockinga.query as blocking_query,
  blockingl.mode as blocking_mode
FROM pg_catalog.pg_locks blockedl
JOIN pg_stat_activity blockeda ON blockedl.pid = blockeda.pid
JOIN pg_catalog.pg_locks blockingl ON(
  ( (blockingl.transactionid=blockedl.transactionid) OR
  (blockingl.relation=blockedl.relation AND blockingl.locktype=blockedl.locktype)
  ) AND blockedl.pid != blockingl.pid)
JOIN pg_stat_activity blockinga ON blockingl.pid = blockinga.pid
  AND blockinga.datid = blockeda.datid
WHERE NOT blockedl.granted
AND blockinga.datname = current_database()

通过这个的结果，可以清楚的看到是哪个 pid 造成了阻塞

locked_item	waiting_duration	blocked_pid	blocked_query	blocked_mode	blocking_pid	blocking_query	blocking_mode
...	00:02:21.268013	8131	alter table  add column ;
	AccessExclusiveLock	5227	<insufficient privilege>	AccessShareLock

接下来，就是处理了，看了一下，那个是个 select，那就不客气了

杀掉有两种方式

select pg_cancel_backend(<pid of the process>)

If the process cannot be killed, try:

select pg_terminate_backend(<pid of the process>)

greenplum 集群启动失败

Fri, 06 Jul 2018 14:39:33 +0800

有一个 gp 集群，部分 seg 挂了，带伤跑了一阵子，执行过几次 gprecovery，又中断了 recovery 进程，最后发现 gprecovery 无法继续，总是会触发节点 down，然后继续作死，把之前备份的一份 datadir 的内容替换上去，再次重启 gp 集群，发现这下好了，彻底起不来，报错

20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-----------------------------------------------------
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-DBID:17  FAILED  host:'10.240.128.21' datadir:'/data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Mirror_seg4_17' with reason:'PG_CTL failed.'
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-DBID:23  FAILED  host:'10.240.128.21' datadir:'/data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Mirror_seg10_23' with reason:'PG_CTL failed.'
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-DBID:6  FAILED  host:'10.240.128.14' datadir:'/data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Primary_seg4_6' with reason:'Failure in segment mirroring; check segment logfile'
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-DBID:12  FAILED  host:'10.240.128.14' datadir:'/data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Primary_seg10_12' with reason:'Failure in segment mirroring; check segment logfile'
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-DBID:3  FAILED  host:'10.240.128.21' datadir:'/data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Primary_seg1_3' with reason:'PG_CTL failed.'
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-DBID:9  FAILED  host:'10.240.128.21' datadir:'/data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Primary_seg7_9' with reason:'PG_CTL failed.'
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-----------------------------------------------------


20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-----------------------------------------------------
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-   Successful segment starts                                                     = 14
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-Failed segment starts, from mirroring connection between primary and mirror   = 2    <<<<<<<<
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-Other failed segment starts                                                   = 4    <<<<<<<<
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-Skipped segment starts (segments are marked down in configuration)            = 2    <<<<<<<<
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-----------------------------------------------------
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-Successfully started 14 of 20 segment instances, skipped 2 other segments <<<<<<<<
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-----------------------------------------------------
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-Segment instance startup failures reported
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-Failed start 6 of 20 segment instances <<<<<<<<
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-Review /data/gprds/gpAdminLogs/gpstart_20180705.log
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-----------------------------------------------------
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-****************************************************************************
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-There are 2 segment(s) marked down in the database
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-To recover from this current state, review usage of the gprecoverseg
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-management utility which will recover failed segment instance databases.
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-****************************************************************************
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-Commencing parallel segment instance shutdown, please wait...
.............

20180705:21:20:11:021616 gpstart:10:gprds-[ERROR]:-gpstart error: Do not have enough valid segments to start the array.

用 gpstart error: Do not have enough valid segments to start the array. 这个错误搜了一下，前几篇没什么收获，继续找，看到这里， http://blog.163.com/digoal@126/blog/static/163877040201122415843868/ 发现大家同样出现 ‘PG_CTL failed.’ 的报错，所以他的解决方案是增加超时时间

去看回 gpstart 的说明文档，也是有类似的阐述

-t | --timeout <number_of_seconds>

  Specifies a timeout in seconds to wait for a segment instance to
  start up. If a segment instance was shutdown abnormally (due to
  power failure or killing its postgres database listener process,
  for example), it may take longer to start up due to the database
  recovery and validation process. If not specified, the default timeout
  is 60 seconds.

不过，他的日志中是明确出现了 database system was not properly shut down; automatic recovery in progress 的提示的

2011-03-24 01:51:49.539239 CST,,,p16689,th-507526368,,,,0,,,seg-1,,,,,"LOG","00000","database system was interrupted at 2011-03-23 18:35:06 CST",,,,,,,0,,"xlog.c",5623,
2011-03-24 01:51:49.627039 CST,,,p16689,th-507526368,,,,0,,,seg-1,,,,,"LOG","00000","checkpoint record is at 111/CFA98080",,,,,,,0,,"xlog.c",5700,
2011-03-24 01:51:49.627083 CST,,,p16689,th-507526368,,,,0,,,seg-1,,,,,"LOG","00000","redo record is at 111/CFA98080; undo record is at 0/0; shutdown FALSE",,,,,,,0,,"xlog.c",5739,
2011-03-24 01:51:49.627099 CST,,,p16689,th-507526368,,,,0,,,seg-1,,,,,"LOG","00000","next transaction ID: 0/21258188; next OID: 595602762",,,,,,,0,,"xlog.c",5743,
2011-03-24 01:51:49.627110 CST,,,p16689,th-507526368,,,,0,,,seg-1,,,,,"LOG","00000","next MultiXactId: 1; next MultiXactOffset: 0",,,,,,,0,,"xlog.c",5746,
2011-03-24 01:51:49.627142 CST,,,p16689,th-507526368,,,,0,,,seg-1,,,,,"LOG","00000","database system was not properly shut down; automatic recovery in progress",,,,,,,0,,"xlog.c",5829,
2011-03-24 01:51:49.627323 CST,,,p16689,th-507526368,,,,0,,,seg-1,,,,,"LOG","00000","redo starts at 111/CFA980D8",,,,,,,0,,"xlog.c",5893,
2011-03-24 01:51:49.729438 CST,"greenplum","postgres",p16690,th-507526368,"[local]",,2011-03-24 01:51:49 CST,0,,,seg-1,,,,,"FATAL","57P03","the database system is starting up",,,,,,,0,,"postmaster.c",1887,"Traceback 0: 0x98615e: /opt/greenplumdb/3.3.6.1/greenplum-db-3.3.6.1/bin/postgres errstart+0x3be
Traceback 1: 0x7c10e8: /opt/greenplumdb/3.3.6.1/greenplum-db-3.3.6.1/bin/postgres +0x7c10e8
Traceback 2: 0x7c2216: /opt/greenplumdb/3.3.6.1/greenplum-db-3.3.6.1/bin/postgres +0x7c2216
Traceback 3: 0x7c35e5: /opt/greenplumdb/3.3.6.1/greenplum-db-3.3.6.1/bin/postgres PostmasterMain+0x945
Traceback 4: 0x6e533b: /opt/greenplumdb/3.3.6.1/greenplum-db-3.3.6.1/bin/postgres main+0x48b
Traceback 5: 0x3f2501d994: /lib64/libc.so.6 __libc_start_main+0xf4
Traceback 6: 0x45c899: /opt/greenplumdb/3.3.6.1/greenplum-db-3.3.6.1/bin/postgres +0x45c899

反观我们的日志，提示的是

[gprds@10 /data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Primary_seg4_6/pg_log]$ cat -n postgresql-21.log
2018-07-05 21:09:45.067408 CST,"gprds","postgres",p46768,th365434944,"[local]",,2018-07-05 21:09:45 CST,0,,,seg-1,,,,,"FATAL","57M01","the database system is in mirror or uninitialized mode",,,,,,,0,,"postmaster.c",2946,
2018-07-05 21:09:45.269644 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","setsockopt(TCP_KEEPCNT) failed: Invalid argument",,,,,,,0,,"pqcomm.c",1953,
2018-07-05 21:09:45.270808 CST,,,p46772,th365434944,"127.0.0.1","8902",2018-07-05 21:09:45 CST,0,,,seg-1,,,,,"LOG","00000","received transition request packet. processing the request",,,,,,,0,,"postmaster.c",2696,
2018-07-05 21:09:45.271015 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: shutting down filerep backends",,,,,,,0,,"primary_mirror_mode.c",1988,
2018-07-05 21:09:45.271044 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep backends to shutdown",,,,,,,0,,"primary_mirror_mode.c",1991,
2018-07-05 21:09:45.271068 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: shutting down filerep",,,,,,,0,,"primary_mirror_mode.c",1998,
2018-07-05 21:09:45.271101 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep to shutdown",,,,,,,0,,"primary_mirror_mode.c",2001,
2018-07-05 21:09:45.271131 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: completed filerep to shutdown",,,,,,,0,,"primary_mirror_mode.c",2004,
2018-07-05 21:09:45.271155 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: initializing XLog Startup",,,,,,,0,,"primary_mirror_mode.c",2011,
2018-07-05 21:09:45.271207 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: starting filerep",,,,,,,0,,"primary_mirror_mode.c",2049,
2018-07-05 21:09:45.271636 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:09:45.271911 CST,,,p46773,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'role not initialized' mirroring state 'not initialized' segment state 'not initialized' filerep state 'not initialized' process name(pid) 'filerep main process(46773)' 'cdbfilerep.c' 'L2598' 'FileRep_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.271954 CST,,,p46773,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'guc 'gp_segment_connect_timeout' value '600' ', mirroring role 'role not initialized' mirroring state 'not initialized' segment state 'not initialized' filerep state 'not initialized' process name(pid) 'filerep main process(46773)' 'cdbfilerep.c' 'L2724' 'FileRep_SetFileRepRetry'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.276103 CST,,,p46773,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'mirror transition, primary address(port) '10.240.128.14(40011)' mirror address(port) '10.240.128.21(40011)' ', mirroring role 'primary role' mirroring state 'sync' segment state 'not initialized' filerep state 'not initialized' process name(pid) 'filerep main process(46773)' 'cdbfilerep.c' 'L3514' 'FileRep_Main'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.276155 CST,,,p46773,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","mirror transition, primary address(port) '10.240.128.14(40011)' mirror address(port) '10.240.128.21(40011)'",,,,,"mirroring role 'primary role' mirroring state 'sync' segment state 'not initialized' process name(pid) 'filerep main process(46773)' filerep state 'not initialized' ",,0,,"cdbfilerep.c",3524,
2018-07-05 21:09:45.277155 CST,,,p46774,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'initialization and recovery' process name(pid) 'primary receiver ack process(46774)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.277210 CST,,,p46774,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'start receiver ack', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary receiver ack process(46774)' 'cdbfilerepprimaryack.c' 'L123' 'FileRepAckPrimary_StartReceiver'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.277255 CST,,,p46774,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'primary address(port) '10.240.128.14(40011)' mirror address(port) '10.240.128.21(40011)' ', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary receiver ack process(46774)' 'cdbfilerepprimaryack.c' 'L134' 'FileRepAckPrimary_StartReceiver'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.277374 CST,,,p46775,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'initialization and recovery' process name(pid) 'primary sender process(46775)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.277441 CST,,,p46775,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'start sender', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary sender process(46775)' 'cdbfilerepprimary.c' 'L1519' 'FileRepPrimary_StartSender'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.277478 CST,,,p46775,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'primary address(port) '10.240.128.14(40011)' mirror address(port) '10.240.128.21(40011)' ', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary sender process(46775)' 'cdbfilerepprimary.c' 'L1530' 'FileRepPrimary_StartSender'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.277520 CST,,,p46776,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'initialization and recovery' process name(pid) 'primary consumer ack process(46776)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.277554 CST,,,p46776,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'run consumer', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary consumer ack process(46776)' 'cdbfilerepprimaryack.c' 'L907' 'FileRepAckPrimary_StartConsumer'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.277713 CST,,,p46777,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'initialization and recovery' process name(pid) 'primary recovery process(46777)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.278881 CST,,,p46777,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'start recovery', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary recovery process(46777)' 'cdbfilerepprimaryrecovery.c' 'L41' 'FileRepPrimary_StartRecovery'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.278924 CST,,,p46777,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'run recovery', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary recovery process(46777)' 'cdbfilerepprimaryrecovery.c' 'L80' 'FileRepPrimary_StartRecoveryInSync'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:10:35.549605 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:11:25.828588 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:12:16.105980 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:13:06.385168 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:13:56.663586 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:14:46.941051 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:15:37.219287 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:16:27.498683 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:17:17.778104 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:18:08.056911 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:18:58.336861 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:19:45.023110 CST,,,p46775,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'initialization and recovery' process name(pid) 'primary sender process(46775)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023196 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"WARNING","01000","PostmasterPrimaryMirrorTransition (2) Finished with Error",,,,,,,0,,"primary_mirror_mode.c",1708,
2018-07-05 21:19:45.023225 CST,,,p46775,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'fault' process name(pid) 'primary sender process(46775)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023258 CST,,,p46775,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'fault' process name(pid) 'primary sender process(46775)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023291 CST,,,p46775,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary sender process(46775)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023325 CST,,,p46776,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'run consumer', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary consumer ack process(46776)' 'cdbfilerepprimaryack.c' 'L966' 'FileRepAckPrimary_RunConsumer'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023360 CST,,,p46777,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'run recovery of flat files', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary recovery process(46777)' 'cdbfilerepprimaryrecovery.c' 'L145' 'FileRepPrimary_RunRecoveryInSync'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023416 CST,,,p46775,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary sender process(46775)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023447 CST,,,p46776,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary consumer ack process(46776)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023477 CST,,,p46776,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary consumer ack process(46776)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023512 CST,,,p46777,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","failure is detected in segment mirroring, failover requested",,,,,"mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' process name(pid) 'primary recovery process(46777)' filerep state 'initialization and recovery' ",,0,,"cdbfilerepprimary.c",265,
2018-07-05 21:19:45.028875 CST,,,p46772,th365434944,"127.0.0.1","8902",2018-07-05 21:09:45 CST,0,,,seg-1,,,,,"WARNING","01000","PrimaryMirrorTransitionRequest (2) Result: Transition to primary/mirror mode PrimarySegment, data state InSync resulted in Error",,,,,,,0,,"primary_mirror_mode.c",1324,
2018-07-05 21:19:45.042936 CST,,,p46774,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary receiver ack process(46774)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.042991 CST,,,p46774,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary receiver ack process(46774)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:58.308632 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","received immediate shutdown request",,,,,,,0,,"postmaster.c",4112,
2018-07-05 21:19:58.309333 CST,,,p46773,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary receiver ack process' process pid '46774' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(46773)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:58.309411 CST,,,p46773,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary consumer ack process' process pid '46776' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(46773)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:58.309485 CST,,,p46773,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary recovery process' process pid '46777' exit status '2' ', mirroring role 'primaryrole' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(46773)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:58.309736 CST,,,p46773,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary sender process' process pid '46775' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(46773)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,

以及

[gprds@10 /data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Primary_seg10_12/pg_log]$ cat -n postgresql-21.log
2018-07-05 21:10:35.566722 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:11:25.844929 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:12:16.122051 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:13:06.399411 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:13:56.675466 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:14:46.953732 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:15:37.230129 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:16:27.505817 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:17:17.782397 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:18:08.060997 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:18:58.339081 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:19:45.028348 CST,,,p46782,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'initialization and recovery' process name(pid) 'primary sender process(46782)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028424 CST,,,p46782,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'fault' process name(pid) 'primary sender process(46782)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028470 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"WARNING","01000","PostmasterPrimaryMirrorTransition (2) Finished with Error",,,,,,,0,,"primary_mirror_mode.c",1708,
2018-07-05 21:19:45.028495 CST,,,p46782,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'fault' process name(pid) 'primary sender process(46782)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028528 CST,,,p46782,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary sender process(46782)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028564 CST,,,p46782,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary sender process(46782)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028596 CST,,,p46783,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'run consumer', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary consumer ack process(46783)' 'cdbfilerepprimaryack.c' 'L966' 'FileRepAckPrimary_RunConsumer'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028630 CST,,,p46784,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'run recovery of flat files', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary recovery process(46784)' 'cdbfilerepprimaryrecovery.c' 'L145' 'FileRepPrimary_RunRecoveryInSync'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028680 CST,,,p46783,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary consumer ack process(46783)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028711 CST,,,p46783,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary consumer ack process(46783)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028746 CST,,,p46784,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","failure is detected in segment mirroring, failover requested",,,,,"mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' process name(pid) 'primary recovery process(46784)' filerep state 'initialization and recovery' ",,0,,"cdbfilerepprimary.c",265,
2018-07-05 21:19:45.037534 CST,,,p46779,th479004736,"127.0.0.1","41654",2018-07-05 21:09:45 CST,0,,,seg-1,,,,,"WARNING","01000","PrimaryMirrorTransitionRequest (2) Result: Transition to primary/mirror mode PrimarySegment, data state InSync resulted in Error",,,,,,,0,,"primary_mirror_mode.c",1324,
2018-07-05 21:19:45.060347 CST,,,p46781,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary receiver ack process(46781)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.060419 CST,,,p46781,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary receiver ack process(46781)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:58.308578 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","received immediate shutdown request",,,,,,,0,,"postmaster.c",4112,
2018-07-05 21:19:58.309359 CST,,,p46780,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary receiver ack process' process pid '46781' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(46780)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:58.309401 CST,,,p46780,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary consumer ack process' process pid '46783' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(46780)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:58.309523 CST,,,p46780,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary recovery process' process pid '46784' exit status '2' ', mirroring role 'primaryrole' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(46780)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:58.309843 CST,,,p46780,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary sender process' process pid '46782' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(46780)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,

其中也有提到 recovery，不过并不是那么的明确

不过死马当作活马医，还是试试在主节点上执行 gpstart -B 1 -t 3600 -v 看看，其中 -B 1 让他串行，-t 3600 把超时置为一个钟，-v 让他打印详细日志

但是依然在启动到 QE_Primary_seg10_12 的时候失败，从报错看，还是类似的报错

2018-07-05 21:56:08.740151 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:56:59.017695 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:57:49.296098 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:58:39.574285 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:59:29.852117 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 22:00:20.129710 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 22:01:10.407472 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 22:02:00.687284 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 22:02:50.967451 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 22:03:41.246303 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 22:04:31.527052 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 22:05:18.008223 CST,,,p59333,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary sender process(59333)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008310 CST,,,p59333,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'fault' process name(pid) 'primary sender process(59333)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008353 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"WARNING","01000","PostmasterPrimaryMirrorTransition (2) Finished with Error",,,,,,,0,,"primary_mirror_mode.c",1708,
2018-07-05 22:05:18.008390 CST,,,p59333,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'fault' process name(pid) 'primary sender process(59333)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008432 CST,,,p59333,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary sender process(59333)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008469 CST,,,p59333,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary sender process(59333)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008506 CST,,,p59337,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'run recovery of flat files', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary recovery process(59337)' 'cdbfilerepprimaryrecovery.c' 'L145' 'FileRepPrimary_RunRecoveryInSync'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008568 CST,,,p59335,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'run consumer', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary consumer ack process(59335)' 'cdbfilerepprimaryack.c' 'L966' 'FileRepAckPrimary_RunConsumer'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008607 CST,,,p59335,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary consumer ack process(59335)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008642 CST,,,p59335,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary consumer ack process(59335)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008682 CST,,,p59328,th-1729120192,"127.0.0.1","53902",2018-07-05 21:55:18 CST,0,,,seg-1,,,,,"WARNING","01000","PrimaryMirrorTransitionRequest (2) Result: Transition to primary/mirror mode PrimarySegment, data state InSync resulted in Error",,,,,,,0,,"primary_mirror_mode.c",1324,
2018-07-05 22:05:18.008717 CST,,,p59337,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","failure is detected in segment mirroring, failover requested",,,,,"mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' process name(pid) 'primary recovery process(59337)' filerep state 'initialization and recovery' ",,0,,"cdbfilerepprimary.c",265,
2018-07-05 22:05:18.028315 CST,,,p59331,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary receiver ack process(59331)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.028373 CST,,,p59331,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary receiver ack process(59331)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:32.044313 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","received immediate shutdown request",,,,,,,0,,"postmaster.c",4112,
2018-07-05 22:05:32.044937 CST,,,p59330,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary consumer ack process' process pid '59335' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(59330)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:32.044984 CST,,,p59330,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary recovery process' process pid '59337' exit status '2' ', mirroring role 'primary role'mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(59330)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:32.045051 CST,,,p59330,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary receiver ack process' process pid '59331' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(59330)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:32.045410 CST,,,p59330,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary sender process' process pid '59333' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(59330)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,

仔细读了一下日志，原因应该是从 filerep state ‘fault’ 这里开始导致的，并因此导致了 data state InSync resulted in Error 最后进程结束退出

拿着这个 filerep state ‘fault’ 搜一下呢？很欣喜的发现来到了无人区，全网只有一条结果 https://github.com/greenplum-db/gpdb/issues/1397 并且还没什么解决方案

于是接下来就开始进入自主探索阶段，首先想到去 QE_Primary_seg10_12 的 Mirror 节点看看，在 /data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Mirror_seg10_23/pg_log 我们从日志中发现这么一个提示

2018-07-05 13:54:14.508270 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""logging_collector"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.508881 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""log_destination"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.509455 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""log_directory"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.509982 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""log_destination"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.510511 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""log_directory"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.511090 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""logging_collector"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.511640 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""log_destination"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.512169 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""log_directory"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.512692 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""log_destination"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.513231 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""log_directory"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.514707 GMT,,,p40050,th63346752,,,,0,,,seg-1,,,,,"FATAL","53100","could not write lock file ""postmaster.pid"": No space left on device",,,,,,,,"CreateLockFile","miscinit.c",1019,1 0x8c9846 postgres errstart + 0x1f6
2    0x8da406 postgres <symbol not found> + 0x8da406
3    0x768d3a postgres PostmasterMain + 0x88a
4    0x488ddb postgres main + 0x3bb
5    0x7f8a0281eb35 libc.so.6 __libc_start_main + 0xf5
6    0x488ef9 postgres <symbol not found> + 0x488ef9

说是磁盘没有空间了，顺手 df -h 看一眼

[gprds@10 /data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Mirror_seg10_23/pg_log]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        20G  3.5G   16G  19% /
devtmpfs         32G     0   32G   0% /dev
tmpfs            32G   16K   32G   1% /dev/shm
tmpfs            32G  3.2G   29G  11% /run
tmpfs            32G     0   32G   0% /sys/fs/cgroup
/dev/sda3        20G  857M   18G   5% /usr/local
/dev/sda4       1.8T   14G  1.7T   1% /data
/dev/md127p1    9.1T  8.6T     0 100% /data1
tmpfs           6.3G     0  6.3G   0% /run/user/0
tmpfs           6.3G     0  6.3G   0% /run/user/1000

我的天，9.1T 的盘被吃的干干净净，这是什么情况

[gprds@10 /data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10]$ du -h --max-depth=1
3.0T	./QE_Primary_seg7_9
78G	./QE_Mirror_seg4_17
5.5T	./QE_Primary_seg1_3
76G	./QE_Mirror_seg10_23
8.6T	.

进去这两个一看发现 core 得尸横遍野

连删都删不掉

[gprds@10 /data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Primary_seg1_3]$ rm core.61416
mkdir: cannot create directory '/data1/.Trash/gprds/2018070522': No space left on device

看起来是 alias 了 rm 命令，但是看了一下 alias 又没有

[gprds@10 /data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Primary_seg1_3]$ alias
alias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'
alias grep='grep --color=auto'
alias l.='ls -d .* --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=auto'
alias vi='vim'
alias which='alias | /usr/bin/which --tty-only --read-alias --show-dot --show-tilde'

不管，先 echo "" > core.61416 清掉一个文件的内容再说，剩下的 core 一个个删除，但是发现删除的东西都被移动到 Trash 里面，空间并没有释放出来

于是尝试用 unlink 来删除 core 文件，发现这个是个好东西，可以把文件删掉，但是比较慢，删除一个 10G 的 core 文件要好一会，试了试 find ./ -name “core.48980” -delete 的方式，也是差不多一样慢，不过好在都可以把文件删掉

腾出来空间之后再次通过 gpstart -B 1 -t 3600 -v 来尝试启动集群，这回总算把集群启动起来了

但是这样并没有解决集群的数据恢复问题，于是冒着作死的精神，又把之前那份备份的 Mirror 节点的 datadir 拷贝过来了

居然可以起来，于是重新开始 gprecoverseg -F 的漫长过程

历史评论

house (2018-08-31 11:27:52):

牛逼哈哈

ZRJ (2018-08-31 19:18:23):

牛个毛线，然而最后并没有恢复成功

一顿操作猛如虎，一看战绩零比五

house (2018-09-01 00:14:11):

我一般就卡在google结果为空的步骤了

postgresql 统计表大小

Fri, 11 May 2018 17:51:04 +0800

分享一个统计 postgresql 表大小的 sql

with table_size_in_bytes as (
    select table_schema
    , table_name
    , pg_size_pretty(pg_total_relation_size(table_schema||'.'||table_name)::bigint) 
        as size_in_humanreadable
    , pg_total_relation_size(table_schema||'.'||table_name) as size_in_byte
    from information_schema.tables 
)
(
    select 'ALL_SCHEMA' as table_schema
    , 'ALL_TABLE' as table_name
    , pg_size_pretty(sum(size_in_byte)::bigint) as size_in_humanreadable
    , sum(size_in_byte) as size_in_byte
    from 
    table_size_in_bytes
)
union all
(
    select table_schema
    , 'ALL_TABLE' as table_name
    , pg_size_pretty(sum(size_in_byte)::bigint) as size_in_humanreadable
    , sum(size_in_byte) as size_in_byte
    from 
    table_size_in_bytes
    group by table_schema
    order by sum(size_in_byte) desc
)
union all
(
    select table_schema
    , table_name
    , size_in_humanreadable
    , size_in_byte
    from 
    table_size_in_bytes
    order by size_in_byte desc
);

postgresql 一些坑

Thu, 30 Nov 2017 00:04:08 +0800

看到这里， https://www.zhihu.com/question/20010554 在总结 pg 的一些好处

但是，个人实际用下来，pg 的两个大坑，一个是 auto vacuum 不生效，慢，而且没用，vacuum full 更慢，而且效果不明显，mvcc 我可以理解，但是不应该空间膨胀来作为代价吧

另外一个，就是慢查询日志，这个貌似还要通过插件来实现，还没细究

=============

2017-11-30 00:15:22 https://yq.aliyun.com/ask/25 这里提到用 log_min_duration_statement 的配置，应该是一个更好的办法

postgresql 表大小控制

Fri, 29 Sep 2017 20:41:10 +0800

如果在构造一张很宽的 postgresql 表的时候，发现表磁盘占用膨胀的很厉害，且这张宽表是一张稀疏表，那么可以把所有的稀疏字段都用 null 赋值（而不是 0）

这个心得貌似很简单，但是花费了我好大周折才总结出来，表空间缩小很显著

spark 写 gp/tpg 效率优化 —— 写入 237w 行数据耗时从 77 分钟到 34 秒

Thu, 27 Jul 2017 09:38:06 +0800

摘自内部分享，有删减。

具体到我们这次的场景中，我们用的是 gp，gp 全称是 greenplum，是一个 mpp 版本的 postgresql，可以参考这个简介 http://www.infoq.com/cn/news/2015/11/PostgreSQL-Pivotal ，协议上兼容 postgresql，我们可以用普通能连 postgresql 的方式去连 gp，并且把 gp 看成一个黑盒的集群版本的 postgresql 来使用

然后这次的优化的手段也很简单，就是从原来的 jdbc 连接拼 sql 改成用 org.postgresql.copy.CopyManager，类似 postgresql 命令行下的 \copy 命令，所以一句话就能说完，而写这个文章的点主要是分享一下这个过程中的一些思路历程和细节

对比图

作为对比，我们原先的数据写入方式是 jdbc 连上之后拼 insert 语句，应该说这种方式在 OLTP 场景下是很适用的，但是在 OLAP 场景下效率问题就开始显现出来了，耗时不仅仅产生在写入端拼 query string 的开销上，更重的是在 db server 端去 parse query 的耗时成本，以及附带衍生的事务，回滚日志等开销成本。

那么 gp 作为一个立足于大量数据处理的 RDBMS，肯定要对数据的 IO 有一个解决方案的，官方是怎么来解决这个问题的呢，看到这里，https://gpdb.docs.pivotal.io/4320/admin_guide/load.html ，官方主要提供了几种方案：

External Tables enable accessing external files as if they are regular database tables.
gpload provides an interface to the Greenplum Database parallel loader.
COPY is the standard PostgreSQL non-parallel data loading tool.

其中，外部表的方式可以通过以下几种实现达成

gpfdist: points to a directory on the file host and serves external data files to all Greenplum Database segments in parallel.
gpfdists: the secure version of gpfdist.
file:// accesses external data files on a segment host that the Greenplum superuser (gpadmin) can access.
gphdfs: accesses files on a Hadoop Distributed File System (HDFS).

gpfdist 可以把一个外部机器上的数据文件让所有 seg 节点能访问到，因而就可以并行的载入数据，gpfdists 是一个安全版本的 gpfdist，但是这种方式存在一些问题考量：

需要在 seg 节点上额外安装部署程序
不兼容 tpg（tpg 本身就没有 seg 节点的概念，囧）

所以没有选用，而 gphdfs 这种方式，能够让 gp 连上 hdfs 去读数据，也是并行的，但是出于以下考量也最终没有采用

我们的 tdw 的 hdfs 带了自定义的鉴权
我们在 hive 表中的存储格式并不是平坦的二维表，由于指标的值稀疏，我们使用的是类似 postgresql 的 hstore 的存储格式，而这种形式并不利于直接表对表的拷到 gp 中成为一张平坦的表

而 gpload 其实就是一个外部表的载入界面封装

The gpload data loading utility is the interface to Greenplum’s external table parallel loading feature.

所以最终我们的选择就落在了 copy 上，用 copy 的好处主要是

是 postgresql 的标准工具，无缝兼容 gp 与 tpg，一次干活到处使用
不需要额外的依赖与安装部署，对目标 server 没有特殊要求

虽然，在官方的介绍中说了 copy 是一个非并行的工具，但是，实测下来，copy 的效率并不低

用 copy 有两种方式，一种是在命令行上用，参考 https://www.postgresql.org/docs/9.2/static/sql-copy.html ，另外一种，是引入 jar 包，在代码中用，参考 https://jdbc.postgresql.org/documentation/publicapi/org/postgresql/copy/CopyManager.html ，可以看到他在函数的 Javadoc 上说了

Use COPY FROM STDIN for very fast copying from an InputStream into a database table.

这也说明这个工具的作者是很自信的（笑）

可以看到这个函数有两个重载

public long copyIn(String sql,
                   Reader from)
            throws SQLException,
                   IOException

和

public long copyIn(String sql,
                   InputStream from)
            throws SQLException,
                   IOException

那么自然的问题就是，从 Reader 中读取和从 InputStream 中读取有什么区别？

参考这个文章， http://blog.sina.com.cn/s/blog_6d3183b50101cri5.html ，其实，二者的主要区别在于：

InputStream提供的是字节流的读取，而非文本读取，这是和Reader类的根本区别。即用Reader读取出来的是char数组或者String ，使用InputStream读取出来的是byte数组。

也就是说，他们一个是面向字节的，一个是面向字符的，而面向字符的自然就要面临一个问题就是字符的编码方式的选择问题，以及解码和编码的开销成本问题，所以从效率上来说，我们应该是选用面向字节的方式

去看他的源码实现也可以发现

    public long copyIn(final String sql, InputStream from, int bufferSize) throws SQLException, IOException {
        byte[] buf = new byte[bufferSize];
        int len;
        CopyIn cp = copyIn(sql);
        try {
            while( (len = from.read(buf)) > 0 ) {
                cp.writeToCopy(buf, 0, len);
            }
            return cp.endCopy();
        } finally { // see to it that we do not leave the connection locked
            if(cp.isActive())
                cp.cancelCopy();
        }
    }

对于 InputStream 的方式，直接读进来就可以用了

    public long copyIn(final String sql, Reader from, int bufferSize) throws SQLException, IOException {
        char[] cbuf = new char[bufferSize];
        int len;
        CopyIn cp = copyIn(sql);
        try {
            while ( (len = from.read(cbuf)) > 0) {
                byte[] buf = encoding.encode(new String(cbuf, 0, len));
                cp.writeToCopy(buf, 0, buf.length);
            }
            return cp.endCopy();
        } finally { // see to it that we do not leave the connection locked
            if(cp.isActive())
                cp.cancelCopy();
        }
    }

而对于 Reader 的方式，读进来之后还要 encoding 处理一下，进一步验证了我们的想法

接下来，就可以得出一个使用方式的 demo 代码了，我们的根本需求是把计算结果写入，所以应该是 RDD[T] ，但是，由于需要转成一个 InputStream，所以我们需要转而接受一个 Array[Array[String]] 的入参

所以到目前为止可以得出代码如下

  def copyIn(data: Array[Array[String]], tblName: String): Long = {
    var con: Connection = null
    try {      
      Class.forName("org.postgresql.Driver")
      println("connecting to database with url " + url)
      con = DriverManager.getConnection(url, user, password)
      val cm = new CopyManager(con.asInstanceOf[BaseConnection])
      val COPY_CMD = s"COPY $tblName from STDIN"
      val start = System.currentTimeMillis()
      val affectedRowCount = cm.copyIn(COPY_CMD, genInputStream(data))
      val finish = System.currentTimeMillis()
      println("copy operation completed successfully in " + (finish-start)/1000.0 + " seconds, affectedRowCount " + affectedRowCount)
      con.close()
      affectedRowCount
    } catch {
      case ex: SQLException => println("Failed to copy data: " + ex.getMessage()); 0
    } finally {
      try {
        if (con != null) con.close()
      } catch {
        case ex: SQLException => println(ex.getMessage())
      }
    }
  }
  
  def genInputStream(arr: Array[Array[String]]): InputStream = {    
    val stringBuilder = new StringBuilder
    println("input data has " + arr.length + " rows")
    if (arr.length != 0) {
      val rowcount = arr.length;
      val columncount = arr(0).length
      for (i <- 0 to rowcount-1; j <- 0 to columncount-1) {
        stringBuilder.append(arr(i)(j) + (if (j == columncount-1) "\r\n" else "\t"))
      }
    }
    val str = stringBuilder.toString
    println("input data " + arr.length + " rows total " + str.length + " bytes")
    new ByteArrayInputStream(str.getBytes(StandardCharsets.UTF_8))
  }

以上这个代码能够正常跑出文章开头的性能测试的结果，但是很显然，埋了一个大坑：会爆内存

且不说为了能够满足 API 的要求，我们需要把输入数据组织成一个 Array[Array[String]] 并拉到 driver 节点（这一步骤本身就违背了 driver 节点不干具体活的宗旨），就说我们把 RDD collect 到 driver 之后，还需要转变出一个 InputStream 这种形式，中途还需要通过 StringBuilder 去 Build 一个大 string，这何止是奢侈，简直就是奢侈，Array[Array[String]] 占一份内存，toString 之后又占一份内存

于是尝试使用 PipedOutputStream 和 PipedInputStream 来解决，这是一个基于管道的流式读写，我们可以起一个单独的线程，来往这个 PipedOutputStream 写入数据，由于缓冲区大小有限，他就会阻塞在缓冲区满的状态下，然后读取端从 PipedInputStream 去读，一边读一边写入到网络上去，jvm 顿时轻松很多，但是，动手之前，有一个问题是，怎么来确认我们的这些改动是真的有效呢，由此需要引入 java 运行时的内存监控工具

我们可以发现，在安装 jdk 的时候，附送了这么一个东西

他其实是一个运行时的监控，简单的 CPU 内存监控可以不需要说明书就能上手

他可以绘制出 jvm 运行时的 cpu 和内存曲线图，并带有仪表盘

另外，我们也通过 runtime 来获取使用的内存，参考这里， http://viralpatel.net/blogs/getting-jvm-heap-size-used-memory-total-memory-using-java-runtime/ 可以加入打印函数如下

  def printMem(currentMoment: String) {
    println(s"=====$currentMoment=========")
    val mb = 1024*1024
    val runtime = Runtime.getRuntime()
    println("Used Memory:" + (runtime.totalMemory() - runtime.freeMemory()) / mb)
    println("Free Memory:" + runtime.freeMemory() / mb)
    println("Total Memory:" + runtime.totalMemory() / mb)
    println("===============")
  }

然后在原有的函数上打点

  def genInputStream(arr: Array[Array[String]]): InputStream = {    
    printMem("before gen string")
    val stringBuilder = new StringBuilder
    println("input data has " + arr.length + " rows")
    if (arr.length != 0) {
      val rowcount = arr.length;
      val columncount = arr(0).length
      for (i <- 0 to rowcount-1; j <- 0 to columncount-1) {
        stringBuilder.append(arr(i)(j) + (if (j == columncount-1) "\r\n" else "\t"))
      }
    }
    val str = stringBuilder.toString
    printMem("after gen string")
    println("input data " + arr.length + " rows total " + str.length + " bytes")
    new ByteArrayInputStream(str.getBytes(StandardCharsets.UTF_8))
  }

并生成 1kw 行测试数据

  def main(args: Array[String]): Unit = {
    //var data = Array(Array("P1","PenDrive","50","US"), Array("P1","PenDrive","300","US"))
    printMem("before gen array")
    val data = Array.fill(100*10000*10)(Array("P1","PenDrive","50","US"))
    printMem("after gen array")
    copyIn(data, "test.product")
  }

我们使用一个测试用的表，直接在 eclipse 中跑一下，可以得到输出如下

=====before gen array=========
Used Memory:3
Free Memory:241
Total Memory:245
===============
=====after gen array=========
Used Memory:345
Free Memory:290
Total Memory:635
===============
=====before gen string=========
Used Memory:352
Free Memory:305
Total Memory:658
===============
input data has 10000000 rows
=====after gen string=========
Used Memory:1989
Free Memory:479
Total Memory:2469
===============
input data 10000000 rows total 190000000 bytes
copy operation completed successfully in 69.951 seconds, affectedRowCount 10000000

可以看到，通过 stringbuilder 来生成 inputstream 的方式，耗用的内存，远比一倍要多

那么接下来就可以尝试改成 PipedOutputStream 和 PipedInputStream 的方式了

把生成 InputStream 的类改成如下

  def genPipedInputStream(arr: Array[Array[String]]): InputStream = {
    printMem("before gen inputstream")
    val out = new PipedOutputStream
    (new Thread(){
      override def run {
        println("input data has " + arr.length + " rows")
        if (arr.length != 0) {
          val rowcount = arr.length;
          val columncount = arr(0).length
          for (i <- 0 to rowcount-1; j <- 0 to columncount-1) {
            out.write((arr(i)(j) + (if (j == columncount-1) "\r\n" else "\t")).getBytes(StandardCharsets.UTF_8))
          }
        }        
        out.close()
        println("PipedOutputStream closed")
      }
    }).start()
    val in = new PipedInputStream
    in.connect(out)
    printMem("after gen inputstream")
    in    
  }

（这里其实隐含了一个问题就是是否需要 CountDownLatch）

可以看到输出如下，耗时有所增加，不过内存控制住了

=====before gen array=========
Used Memory:3
Free Memory:241
Total Memory:245
===============
=====after gen array=========
Used Memory:345
Free Memory:295
Total Memory:641
===============
=====before gen inputstream=========
Used Memory:352
Free Memory:288
Total Memory:641
===============
=====after gen inputstream=========
Used Memory:352
Free Memory:286
input data has 10000000 rows
Total Memory:641
===============
PipedOutputStream closed
copy operation completed successfully in 97.917 seconds, affectedRowCount 10000000

并且过程中的内存曲线基本平稳

接下来，浮现出来的一个问题就是：是否真的需要把 RDD[T] collect 到 driver 上来？

答案其实是可以不需要，我们有 mapPartitions 这个算子，可以写成如下

val start = System.currentTimeMillis()
dataGpFlatten.mapPartitions(x => {
  GPCopyMgr.copyIn(x.toArray, "xxxxx")
  x
}).count
val finish = System.currentTimeMillis()
println("operation completed successfully in " + (finish-start)/1000.0 + " seconds")

需要注意的是，mapPartitions 并不是 action，而是一个 transform，所以我们需要在后面给他跟上一个 action，例如 count，来触发执行

主节点再无写入数据的动作，并且总的耗时比文章开头的耗时还要下降了 5s，不过基本在一个量级，可以认为是实验误差范围内

通过这种 mapPartitions 的方式，需要注意的问题有

partition 数量的选择，过多容易造成同时连接 db 的连接数过多，而且每个分区小了，其实吞吐性能不利
如果需要 re-partition，需要意识到 re-partition 也是有开销成本的
最后别忘了跟一个 action

至此，基本就完结了，剩下就是一些工程化方面的工作，例如

在写入数据之前删除分区，以避免脏数据
在写入数据之后校验写入行数是否相符，以免某个 partition 写的过程中出异常了（这里其实引申出来一个问题，如果某个 executor 在写到一半的时候挂了，怎么办，是否只能整个 lz 任务重跑来清理现场？）
加强日志的可读性

以上动作都是工程化方面的工作，其实还是避免自己给自己挖坑，哈哈

历史评论

郭祥汝 (2019-08-24 12:44:03):

你好，我有两个表，同样是200万行数据，第一张表一条数据不大，第二张表每条数据很大，打一张表copy写入很快，第二张很慢，而且爆出 read and dead错误，请问这是什么问题?

ZRJ (2019-08-29 10:19:15):

这个错误我倒是没有遇到过，搜了下貌似也没有什么资料，有详细的上下文和错误提示吗

郭祥汝 (2019-08-30 17:57:23):

那个表中，每一条数据都是那种大文本数据，这种用copy方式写入数据，很慢，也能写入，时间消耗很长，300万条数据，大概消耗7分钟左右。如果表中不包含大文本数据，300万条30多秒，这个差别很大啊，你有没有其他的建议，谢谢！

ZRJ (2019-09-01 11:42:46):

看下大文本的体积呢，这些大文本每行多少个 byte？同时看下写入时，db 机器的磁盘 IO 和网卡 IO

claude (2026-03-13 14:42:12):

好文章，从 JDBC insert 到 COPY 协议，思路清晰，方案选型也很务实。几个点想交流一下：

关于 PipedInputStream/PipedOutputStream 的方案，这个设计本质上是把「数据序列化」和「网络传输」两个阶段从串行变成了流水线并行，用内存换时间的经典思路。不过 PipedStream 在 JVM 中有个已知的坑——默认 buffer 只有 1024 bytes，如果生产端和消费端速率差异大容易触发频繁的 wait/notify，可以考虑在构造时传一个更大的 bufferSize（比如 64KB），或者干脆用 java.nio 的 Pipe.open() 替代，吞吐更好。

你提到的 executor 写到一半挂掉的问题，其实是分布式写入中很经典的 exactly-once 难题。一个可行的工程方案是：每个 partition 先写到一张带 batch_id 的临时表（或者用 GP 的临时表），全部 partition 成功后再做一次 INSERT INTO … SELECT 或者 ALTER TABLE EXCHANGE PARTITION 来原子性地「翻牌」。这样任何一个 partition 失败都可以安全重试，不用整个任务重跑。

另外，既然用了 mapPartitions，其实可以更进一步——在 mapPartitions 内部直接构造 PipedOutputStream 往 CopyManager 灌数据，连 Array[Array[String]] 的中间物化都省掉了，iterator 直接流式写入，内存开销可以降到几乎为零。