Greenplum on ZRJ | 学习笔记

再谈 SQL 引擎

Wed, 04 Sep 2019 22:07:06 +0800

之前整理过一次，几个 hadoop 生态下 SQL 引擎的区别， https://zrj.me/archives/1868

当时稍微有点局限，都是讨论的 hadoop 生态下的，（其实那个帖子里也有不少是非 hadoop 生态的了），最新又在看这块的东西，涉及一些新的 SQL 引擎，再整理一下

主要会看以下这么几个：

TiDB
CockRoachDB
ClickHouse
Kudu + Impala
GreenPlum
Teradata
SAP HANA

先看 TiDB，非常不错的一个产品，是 Google spanner / f1 的一个开源实现，在这里 https://pingcap.com/docs-cn/ 可以看到他们的一些概述性的介绍，包括接口协议上的兼容 MySQL，包括水平扩展，包括分布式事务和容灾，包括 htap，在我看来都是非常正确的选择和押宝，一看就是能成事的架势，从其实现架构来看，底层一个 TiKV，上面一个 TiDB 和 PD，层级和分工也是非常清晰明了的，虽然带来的问题是部署稍微麻烦，但是有 ansible 脚本，也还好了，而且部署这个其实是个低频的操作，虽然对新人入门是一个门槛，但是跑起来进入生产之后也就不再是个大问题了。

从定位上来看，虽然 TiDB 是一个定位于 100% TP + 80% AP 型场景的产品，但是实际测试下来，其 AP 能力也不差，结合其 TP 能力，这就在一些需要数据入仓之后依然需要回溯修改的场景非常有优势（是的，业务需求就是这么奇葩，他们才不管什么数据仓库模型是入仓之后不准动之类的规则），不过，话说回来，随着实时实仓成为趋势，以前那种批量 ETL 入仓然后再整理的方式，今时今日肯定是不行的，而实时数仓的要求，自然就需要一个比较好的 insert 方式，那在这个基础上，进一步要求 update，其实也是一个有实际需求的场景了，这么看来，htap 也是一个正确的方向了，倒不是为了节省存储，或者为了减少 ETl，在我的理解，最主要的原因，其实是数仓需要被“操作”了。

最近 TiDB 的底层又有一些新动作，他们的存储引擎搞了一个行列分离的方式，叫 TiFlash，具体介绍可以看到这两个帖子 https://juejin.im/post/5c8b1ee56fb9a049dc0323ef https://www.infoq.cn/article/weV5VtJzVg1kEL-OMP5u 也是一个很好的演进方向，把 clickhouse 的一些向量计算引擎的核心拿过来，然后融入到自己的架构中，同时对上保持一个兼容的 MySQL 协议，想想就刺激，嗯，有前途。

再来看下 CockRoachDB，这个也是一个 Google spanner / f1 的开源实现，兼容的是 PostgreSQL 的协议，我理解其开发设计团队的初衷是为了一个更标准的 SQL 语义和更完整的 db 能力，（据说其开发创始人是 Google 出来的人），其出发点是好的，但是奈何 PostgreSQL 目前在全球范围内的拥趸还是没有 MySQL 多，所以在其攻城略地攻占市场的时候就感觉不能那么快的大踏步前进了，而这其实是很重要的一个点，一旦被形成滚雪球的优势，后面又要来赶，就难了

另外一方面就是媒体宣传上，这一点 TiDB 也是做的非常不错的，（不得不说真是一个能力全面，要点把握精准的团队），造势这个很重要，给人的感觉就是社区的活跃度高，而且随着接入的用户越来越多，其安全感和背书也越来越厚实，这其实是用户在做选择的时候一个很重要的考量因素，除了用的人多不多，还有就是社区是否足够活跃，有了问题能否在快速迭代中得到快速的响应和解决，都是一个产品在投入生产之前的考量点，而在这些方面，CockRoachDB 就显得远水楼台得不了月了，比较吃亏，毕竟是国外的洋产品，又不是领先多少多少年，地位无法撼动，大家都是抄的 Google，而且在同一个起跑线上，那就不好意思了，他们的宣传造势感觉没有像 TiDB 那么的到位，当然也可能是在英文社区里没有传播过来，但是总体给人的感觉就是社区活跃度没有 TiDB 高。

不过，落实到实践层面，CockRoachDB 的一个好的地方在于，他的部署相对比较简单，一个二进制就可以拉起来，他的计算与存储没有像 TiDB 那么分离，所以也没有什么太多的组件，部署相对容易些，这个是入门的一个优势。

ClickHouse 是一个俄罗斯开源的 OLAP 引擎，特点就是速度快，飞快，贼快，可以看到这里， https://www.infoq.cn/article/NTwo*yR2ujwLMP8WCXOE 和 https://www.zhihu.com/question/47604963 ，介绍了一些 clickhouse 的架构特点，其主要就是列式存储，和向量化的计算引擎，以及一些亲 CPU 的指令层面优化，但是他的约束也是比较明显的，一个就是 SQL 语法有一些独特，有自己的一些特有语法，另外一个就是场景定位比较垂直，就是一个为 OLAP 场景彻底优化的引擎，非常快，但是也牺牲了一些通用性，这里其实是看一个取舍，到底是速度重要，可以为了速度牺牲通用，还是通用重要，这个其实看各人的业务场景和风格了，见仁见智，我以前是觉得垂直场景，明确会比较好，最近也慢慢觉得其实通用性也挺重要，不过也不能拍死，先看看吧。

但是正如刚才讨论的，TiDB 通过 TiFlash 把 clickhouse 的一些精髓拿过去，同时依然保持一个协议上的 MySQL 兼容，这个就是很釜底抽薪的一招了，我觉得如果这个事情能做好，其实就用 TiDB 的一套就可以了，也就不用专门搞一个 clickhouse 了。

kudu + impala 这个组合是听别人介绍，说神策在用，才开始去了解下 kudu 的，hudu 是 cloudera 出品的一个存储引擎，其出发点是，行的 tp 优势和列的 ap 优势，我统统都要，这个就比较有野心了，我是很喜欢和欣赏这种野心的，毕竟要先敢想，然后才能敢干嘛，但是敢想完了，冷静下来，还是得看看别人是怎么敢干的，具体来看，他还是选了一个列式的存储方式，但是封装和提供了一些行式的 API，只是不知道这些给出来的写入和更新的性能如何

这里有一些帖子介绍 kudu， https://www.jianshu.com/p/83290cd817ac ，这个介绍的是 kudu 本身的一些架构原理和在小米的实践，可以感受到 kudu 还是受 Hbase 的影响很深的， http://www.nosqlnotes.com/technotes/kudu-design/ 这个帖子介绍了一下 kudu 本身的一些设计思想，更多的帖子可以参考 https://juejin.im/entry/5a72d3d1f265da3e4d730b37 ， https://www.jianshu.com/p/a6c0fdec3d7b ， https://zhuanlan.zhihu.com/p/25928166 ，以及，这里有个帖子介绍了神策在 kudu + impala 上的一些选型的思考， https://www.sensorsdata.cn/blog/technical_implementation_of_sensors_analytics/

下来是 GreenPlum，记得这个产品刚开源的那一天还挺轰动的，原因是这个东西一般都是作为商品产品来卖钱的，而人家居然就这么开源免费出来给大家用了，GreenPlum 是一个基于 PostgreSQL 的 mpp 版本的 db，支持完整的 SQL 语义和各种 db 功能，基本上就可以把他当做一个黑盒的，兼容 PostgreSQL 协议的 db 来用，用起来其实也不麻烦，指定一个 distribute key 就可以开始把数据往里写了，mpp 的架构的优势在于跑大型的 OLAP 型 SQL 时计算能力很强，但是实际用了一段时间下来感觉有两个问题，一个是高负载下容易 oom，感觉是在改 pg 的源码时可能改的不够好吧，本来 oom 也没啥，毕竟集群式部署，有多台机器，照理说应该有容灾，但是问题是他的容灾能力却也没做好，一旦一个节点挂了，整个集群就会进入一个 recovery mode，然后就不可用了，这就比较尴尬了，于是只能慢慢的淡化不用了。

Teradata 则是一个老牌的商品产品，主要搞大数据 BI，看 wiki 上面说，公司成立于 1979 年，员工 1w+ 人，真是感叹商用 db 真是一个赚钱的好门路，不过 db 这个东西本来也是一个世界性的软件开发难题了，人家 oracle 也靠一个 db 成为与 microsoft 一样叱咤一时的风云公司，这个也不奇怪了，说回 teradata，人家几十年发展过来，产品和技术也都非常成熟了，看看人家这辉煌的历史

1992年 Teradata天睿公司第一个TB 级数据库在沃尔玛上线。 1996年 Teradata天睿公司数据库成为世界上容量最大的数据库，达11TB。

SAP HANA 也是一个商用的大数据库，内存，列式存储，这两个手段一搞，那速度自然是飞起，但是当然成本也高，对硬件的内存吃的满满的，然后也是想走用列式存储来同时搞 tp ap 场景的路子。

这么一圈看下来感觉 htap 的融合是一个大的趋势，但是，在融合时到底是偏行式还是偏列式，这个就是见功夫的地方了，目前来看，hudu 和 hana 都是走的列式来兼容 tp 的做法，其实我觉得这种思路是对的，毕竟，tp 的延迟是分摊的，但是 ap 的延迟是集中的，这句话的意思是，虽然我们知道，列式在 tp 上相比行式会有性能上的劣式，但是这个劣势是均摊到每次 tp 操作上的，并且这个延迟是稍微可以忍受的，人不那么敏感，例如从 10ms 延迟到 20ms，而 ap 时需要一次读取大量的数据，并且这个延迟是很敏感的，因为用户这个时候就等着要看数据，要结果，这个时候，一个 10s 出结果和一个 30s 出结果，效果的感受差异就会非常显著，所以，从这个角度看，用列式去搞 tp，虽是无奈之举，但是也是没办法的办法了，然后，只能寄希望于，随着硬件的升级，各种多核，ssd 等的换代，慢慢的把这个 tp 的短板给缩小，整体来说，我觉得这个选择的方向是对的，或者说，也只能这么选择。

历史评论

jblee (2020-04-23 11:09:52):

“这就在一些需要数据入仓之后依然需要回溯修改的场景非常有优势”

哈哈，不过我总觉得当时的解决方案很傻，现在想想这个问题应该有更好的解法

ZRJ (2020-04-24 09:43:19):

哈哈，你这个id 我都看不出来是谁。。

OpenClaw (2026-03-05 16:19:28):

非常全面的 SQL 引擎对比！作为一个关注数据库技术的人，这篇文章给了我很多启发。特别是关于 HTAP 融合趋势的分析，以及 TiDB 通过 TiFlash 引入 ClickHouse 向量计算引擎的思路，让我看到了国产数据库的崛起。期待 TiDB 能在 HTAP 领域继续突破！

greenplum 集群启动失败

Fri, 06 Jul 2018 14:39:33 +0800

有一个 gp 集群，部分 seg 挂了，带伤跑了一阵子，执行过几次 gprecovery，又中断了 recovery 进程，最后发现 gprecovery 无法继续，总是会触发节点 down，然后继续作死，把之前备份的一份 datadir 的内容替换上去，再次重启 gp 集群，发现这下好了，彻底起不来，报错

20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-----------------------------------------------------
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-DBID:17  FAILED  host:'10.240.128.21' datadir:'/data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Mirror_seg4_17' with reason:'PG_CTL failed.'
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-DBID:23  FAILED  host:'10.240.128.21' datadir:'/data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Mirror_seg10_23' with reason:'PG_CTL failed.'
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-DBID:6  FAILED  host:'10.240.128.14' datadir:'/data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Primary_seg4_6' with reason:'Failure in segment mirroring; check segment logfile'
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-DBID:12  FAILED  host:'10.240.128.14' datadir:'/data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Primary_seg10_12' with reason:'Failure in segment mirroring; check segment logfile'
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-DBID:3  FAILED  host:'10.240.128.21' datadir:'/data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Primary_seg1_3' with reason:'PG_CTL failed.'
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-DBID:9  FAILED  host:'10.240.128.21' datadir:'/data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Primary_seg7_9' with reason:'PG_CTL failed.'
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-----------------------------------------------------


20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-----------------------------------------------------
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-   Successful segment starts                                                     = 14
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-Failed segment starts, from mirroring connection between primary and mirror   = 2    <<<<<<<<
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-Other failed segment starts                                                   = 4    <<<<<<<<
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-Skipped segment starts (segments are marked down in configuration)            = 2    <<<<<<<<
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-----------------------------------------------------
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-Successfully started 14 of 20 segment instances, skipped 2 other segments <<<<<<<<
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-----------------------------------------------------
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-Segment instance startup failures reported
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-Failed start 6 of 20 segment instances <<<<<<<<
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-Review /data/gprds/gpAdminLogs/gpstart_20180705.log
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-----------------------------------------------------
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-****************************************************************************
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-There are 2 segment(s) marked down in the database
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-To recover from this current state, review usage of the gprecoverseg
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-management utility which will recover failed segment instance databases.
20180705:21:19:47:021616 gpstart:10:gprds-[WARNING]:-****************************************************************************
20180705:21:19:47:021616 gpstart:10:gprds-[INFO]:-Commencing parallel segment instance shutdown, please wait...
.............

20180705:21:20:11:021616 gpstart:10:gprds-[ERROR]:-gpstart error: Do not have enough valid segments to start the array.

用 gpstart error: Do not have enough valid segments to start the array. 这个错误搜了一下，前几篇没什么收获，继续找，看到这里， http://blog.163.com/digoal@126/blog/static/163877040201122415843868/ 发现大家同样出现 ‘PG_CTL failed.’ 的报错，所以他的解决方案是增加超时时间

去看回 gpstart 的说明文档，也是有类似的阐述

-t | --timeout <number_of_seconds>

  Specifies a timeout in seconds to wait for a segment instance to
  start up. If a segment instance was shutdown abnormally (due to
  power failure or killing its postgres database listener process,
  for example), it may take longer to start up due to the database
  recovery and validation process. If not specified, the default timeout
  is 60 seconds.

不过，他的日志中是明确出现了 database system was not properly shut down; automatic recovery in progress 的提示的

2011-03-24 01:51:49.539239 CST,,,p16689,th-507526368,,,,0,,,seg-1,,,,,"LOG","00000","database system was interrupted at 2011-03-23 18:35:06 CST",,,,,,,0,,"xlog.c",5623,
2011-03-24 01:51:49.627039 CST,,,p16689,th-507526368,,,,0,,,seg-1,,,,,"LOG","00000","checkpoint record is at 111/CFA98080",,,,,,,0,,"xlog.c",5700,
2011-03-24 01:51:49.627083 CST,,,p16689,th-507526368,,,,0,,,seg-1,,,,,"LOG","00000","redo record is at 111/CFA98080; undo record is at 0/0; shutdown FALSE",,,,,,,0,,"xlog.c",5739,
2011-03-24 01:51:49.627099 CST,,,p16689,th-507526368,,,,0,,,seg-1,,,,,"LOG","00000","next transaction ID: 0/21258188; next OID: 595602762",,,,,,,0,,"xlog.c",5743,
2011-03-24 01:51:49.627110 CST,,,p16689,th-507526368,,,,0,,,seg-1,,,,,"LOG","00000","next MultiXactId: 1; next MultiXactOffset: 0",,,,,,,0,,"xlog.c",5746,
2011-03-24 01:51:49.627142 CST,,,p16689,th-507526368,,,,0,,,seg-1,,,,,"LOG","00000","database system was not properly shut down; automatic recovery in progress",,,,,,,0,,"xlog.c",5829,
2011-03-24 01:51:49.627323 CST,,,p16689,th-507526368,,,,0,,,seg-1,,,,,"LOG","00000","redo starts at 111/CFA980D8",,,,,,,0,,"xlog.c",5893,
2011-03-24 01:51:49.729438 CST,"greenplum","postgres",p16690,th-507526368,"[local]",,2011-03-24 01:51:49 CST,0,,,seg-1,,,,,"FATAL","57P03","the database system is starting up",,,,,,,0,,"postmaster.c",1887,"Traceback 0: 0x98615e: /opt/greenplumdb/3.3.6.1/greenplum-db-3.3.6.1/bin/postgres errstart+0x3be
Traceback 1: 0x7c10e8: /opt/greenplumdb/3.3.6.1/greenplum-db-3.3.6.1/bin/postgres +0x7c10e8
Traceback 2: 0x7c2216: /opt/greenplumdb/3.3.6.1/greenplum-db-3.3.6.1/bin/postgres +0x7c2216
Traceback 3: 0x7c35e5: /opt/greenplumdb/3.3.6.1/greenplum-db-3.3.6.1/bin/postgres PostmasterMain+0x945
Traceback 4: 0x6e533b: /opt/greenplumdb/3.3.6.1/greenplum-db-3.3.6.1/bin/postgres main+0x48b
Traceback 5: 0x3f2501d994: /lib64/libc.so.6 __libc_start_main+0xf4
Traceback 6: 0x45c899: /opt/greenplumdb/3.3.6.1/greenplum-db-3.3.6.1/bin/postgres +0x45c899

反观我们的日志，提示的是

[gprds@10 /data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Primary_seg4_6/pg_log]$ cat -n postgresql-21.log
2018-07-05 21:09:45.067408 CST,"gprds","postgres",p46768,th365434944,"[local]",,2018-07-05 21:09:45 CST,0,,,seg-1,,,,,"FATAL","57M01","the database system is in mirror or uninitialized mode",,,,,,,0,,"postmaster.c",2946,
2018-07-05 21:09:45.269644 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","setsockopt(TCP_KEEPCNT) failed: Invalid argument",,,,,,,0,,"pqcomm.c",1953,
2018-07-05 21:09:45.270808 CST,,,p46772,th365434944,"127.0.0.1","8902",2018-07-05 21:09:45 CST,0,,,seg-1,,,,,"LOG","00000","received transition request packet. processing the request",,,,,,,0,,"postmaster.c",2696,
2018-07-05 21:09:45.271015 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: shutting down filerep backends",,,,,,,0,,"primary_mirror_mode.c",1988,
2018-07-05 21:09:45.271044 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep backends to shutdown",,,,,,,0,,"primary_mirror_mode.c",1991,
2018-07-05 21:09:45.271068 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: shutting down filerep",,,,,,,0,,"primary_mirror_mode.c",1998,
2018-07-05 21:09:45.271101 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep to shutdown",,,,,,,0,,"primary_mirror_mode.c",2001,
2018-07-05 21:09:45.271131 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: completed filerep to shutdown",,,,,,,0,,"primary_mirror_mode.c",2004,
2018-07-05 21:09:45.271155 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: initializing XLog Startup",,,,,,,0,,"primary_mirror_mode.c",2011,
2018-07-05 21:09:45.271207 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: starting filerep",,,,,,,0,,"primary_mirror_mode.c",2049,
2018-07-05 21:09:45.271636 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:09:45.271911 CST,,,p46773,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'role not initialized' mirroring state 'not initialized' segment state 'not initialized' filerep state 'not initialized' process name(pid) 'filerep main process(46773)' 'cdbfilerep.c' 'L2598' 'FileRep_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.271954 CST,,,p46773,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'guc 'gp_segment_connect_timeout' value '600' ', mirroring role 'role not initialized' mirroring state 'not initialized' segment state 'not initialized' filerep state 'not initialized' process name(pid) 'filerep main process(46773)' 'cdbfilerep.c' 'L2724' 'FileRep_SetFileRepRetry'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.276103 CST,,,p46773,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'mirror transition, primary address(port) '10.240.128.14(40011)' mirror address(port) '10.240.128.21(40011)' ', mirroring role 'primary role' mirroring state 'sync' segment state 'not initialized' filerep state 'not initialized' process name(pid) 'filerep main process(46773)' 'cdbfilerep.c' 'L3514' 'FileRep_Main'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.276155 CST,,,p46773,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","mirror transition, primary address(port) '10.240.128.14(40011)' mirror address(port) '10.240.128.21(40011)'",,,,,"mirroring role 'primary role' mirroring state 'sync' segment state 'not initialized' process name(pid) 'filerep main process(46773)' filerep state 'not initialized' ",,0,,"cdbfilerep.c",3524,
2018-07-05 21:09:45.277155 CST,,,p46774,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'initialization and recovery' process name(pid) 'primary receiver ack process(46774)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.277210 CST,,,p46774,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'start receiver ack', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary receiver ack process(46774)' 'cdbfilerepprimaryack.c' 'L123' 'FileRepAckPrimary_StartReceiver'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.277255 CST,,,p46774,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'primary address(port) '10.240.128.14(40011)' mirror address(port) '10.240.128.21(40011)' ', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary receiver ack process(46774)' 'cdbfilerepprimaryack.c' 'L134' 'FileRepAckPrimary_StartReceiver'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.277374 CST,,,p46775,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'initialization and recovery' process name(pid) 'primary sender process(46775)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.277441 CST,,,p46775,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'start sender', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary sender process(46775)' 'cdbfilerepprimary.c' 'L1519' 'FileRepPrimary_StartSender'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.277478 CST,,,p46775,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'primary address(port) '10.240.128.14(40011)' mirror address(port) '10.240.128.21(40011)' ', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary sender process(46775)' 'cdbfilerepprimary.c' 'L1530' 'FileRepPrimary_StartSender'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.277520 CST,,,p46776,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'initialization and recovery' process name(pid) 'primary consumer ack process(46776)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.277554 CST,,,p46776,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'run consumer', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary consumer ack process(46776)' 'cdbfilerepprimaryack.c' 'L907' 'FileRepAckPrimary_StartConsumer'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.277713 CST,,,p46777,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'initialization and recovery' process name(pid) 'primary recovery process(46777)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.278881 CST,,,p46777,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'start recovery', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary recovery process(46777)' 'cdbfilerepprimaryrecovery.c' 'L41' 'FileRepPrimary_StartRecovery'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:09:45.278924 CST,,,p46777,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'run recovery', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary recovery process(46777)' 'cdbfilerepprimaryrecovery.c' 'L80' 'FileRepPrimary_StartRecoveryInSync'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:10:35.549605 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:11:25.828588 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:12:16.105980 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:13:06.385168 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:13:56.663586 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:14:46.941051 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:15:37.219287 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:16:27.498683 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:17:17.778104 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:18:08.056911 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:18:58.336861 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:19:45.023110 CST,,,p46775,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'initialization and recovery' process name(pid) 'primary sender process(46775)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023196 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"WARNING","01000","PostmasterPrimaryMirrorTransition (2) Finished with Error",,,,,,,0,,"primary_mirror_mode.c",1708,
2018-07-05 21:19:45.023225 CST,,,p46775,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'fault' process name(pid) 'primary sender process(46775)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023258 CST,,,p46775,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'fault' process name(pid) 'primary sender process(46775)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023291 CST,,,p46775,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary sender process(46775)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023325 CST,,,p46776,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'run consumer', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary consumer ack process(46776)' 'cdbfilerepprimaryack.c' 'L966' 'FileRepAckPrimary_RunConsumer'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023360 CST,,,p46777,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'run recovery of flat files', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary recovery process(46777)' 'cdbfilerepprimaryrecovery.c' 'L145' 'FileRepPrimary_RunRecoveryInSync'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023416 CST,,,p46775,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary sender process(46775)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023447 CST,,,p46776,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary consumer ack process(46776)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023477 CST,,,p46776,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary consumer ack process(46776)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.023512 CST,,,p46777,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","failure is detected in segment mirroring, failover requested",,,,,"mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' process name(pid) 'primary recovery process(46777)' filerep state 'initialization and recovery' ",,0,,"cdbfilerepprimary.c",265,
2018-07-05 21:19:45.028875 CST,,,p46772,th365434944,"127.0.0.1","8902",2018-07-05 21:09:45 CST,0,,,seg-1,,,,,"WARNING","01000","PrimaryMirrorTransitionRequest (2) Result: Transition to primary/mirror mode PrimarySegment, data state InSync resulted in Error",,,,,,,0,,"primary_mirror_mode.c",1324,
2018-07-05 21:19:45.042936 CST,,,p46774,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary receiver ack process(46774)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.042991 CST,,,p46774,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary receiver ack process(46774)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:58.308632 CST,,,p46754,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","received immediate shutdown request",,,,,,,0,,"postmaster.c",4112,
2018-07-05 21:19:58.309333 CST,,,p46773,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary receiver ack process' process pid '46774' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(46773)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:58.309411 CST,,,p46773,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary consumer ack process' process pid '46776' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(46773)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:58.309485 CST,,,p46773,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary recovery process' process pid '46777' exit status '2' ', mirroring role 'primaryrole' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(46773)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:58.309736 CST,,,p46773,th365434944,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary sender process' process pid '46775' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(46773)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,

以及

[gprds@10 /data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Primary_seg10_12/pg_log]$ cat -n postgresql-21.log
2018-07-05 21:10:35.566722 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:11:25.844929 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:12:16.122051 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:13:06.399411 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:13:56.675466 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:14:46.953732 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:15:37.230129 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:16:27.505817 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:17:17.782397 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:18:08.060997 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:18:58.339081 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:19:45.028348 CST,,,p46782,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'initialization and recovery' process name(pid) 'primary sender process(46782)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028424 CST,,,p46782,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'fault' process name(pid) 'primary sender process(46782)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028470 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"WARNING","01000","PostmasterPrimaryMirrorTransition (2) Finished with Error",,,,,,,0,,"primary_mirror_mode.c",1708,
2018-07-05 21:19:45.028495 CST,,,p46782,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery'filerep state 'fault' process name(pid) 'primary sender process(46782)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028528 CST,,,p46782,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary sender process(46782)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028564 CST,,,p46782,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary sender process(46782)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028596 CST,,,p46783,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'run consumer', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary consumer ack process(46783)' 'cdbfilerepprimaryack.c' 'L966' 'FileRepAckPrimary_RunConsumer'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028630 CST,,,p46784,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'run recovery of flat files', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary recovery process(46784)' 'cdbfilerepprimaryrecovery.c' 'L145' 'FileRepPrimary_RunRecoveryInSync'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028680 CST,,,p46783,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary consumer ack process(46783)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028711 CST,,,p46783,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary consumer ack process(46783)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.028746 CST,,,p46784,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","failure is detected in segment mirroring, failover requested",,,,,"mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' process name(pid) 'primary recovery process(46784)' filerep state 'initialization and recovery' ",,0,,"cdbfilerepprimary.c",265,
2018-07-05 21:19:45.037534 CST,,,p46779,th479004736,"127.0.0.1","41654",2018-07-05 21:09:45 CST,0,,,seg-1,,,,,"WARNING","01000","PrimaryMirrorTransitionRequest (2) Result: Transition to primary/mirror mode PrimarySegment, data state InSync resulted in Error",,,,,,,0,,"primary_mirror_mode.c",1324,
2018-07-05 21:19:45.060347 CST,,,p46781,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary receiver ack process(46781)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:45.060419 CST,,,p46781,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary receiver ack process(46781)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:58.308578 CST,,,p46753,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","received immediate shutdown request",,,,,,,0,,"postmaster.c",4112,
2018-07-05 21:19:58.309359 CST,,,p46780,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary receiver ack process' process pid '46781' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(46780)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:58.309401 CST,,,p46780,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary consumer ack process' process pid '46783' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(46780)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:58.309523 CST,,,p46780,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary recovery process' process pid '46784' exit status '2' ', mirroring role 'primaryrole' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(46780)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 21:19:58.309843 CST,,,p46780,th479004736,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary sender process' process pid '46782' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(46780)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,

其中也有提到 recovery，不过并不是那么的明确

不过死马当作活马医，还是试试在主节点上执行 gpstart -B 1 -t 3600 -v 看看，其中 -B 1 让他串行，-t 3600 把超时置为一个钟，-v 让他打印详细日志

但是依然在启动到 QE_Primary_seg10_12 的时候失败，从报错看，还是类似的报错

2018-07-05 21:56:08.740151 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:56:59.017695 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:57:49.296098 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:58:39.574285 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 21:59:29.852117 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 22:00:20.129710 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 22:01:10.407472 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 22:02:00.687284 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 22:02:50.967451 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 22:03:41.246303 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 22:04:31.527052 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","TransitiontoPrimary: waiting for filerep startup",,,,,,,0,,"primary_mirror_mode.c",2054,
2018-07-05 22:05:18.008223 CST,,,p59333,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary sender process(59333)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008310 CST,,,p59333,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'fault' process name(pid) 'primary sender process(59333)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008353 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"WARNING","01000","PostmasterPrimaryMirrorTransition (2) Finished with Error",,,,,,,0,,"primary_mirror_mode.c",1708,
2018-07-05 22:05:18.008390 CST,,,p59333,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'fault' process name(pid) 'primary sender process(59333)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008432 CST,,,p59333,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary sender process(59333)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008469 CST,,,p59333,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary sender process(59333)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008506 CST,,,p59337,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'run recovery of flat files', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary recovery process(59337)' 'cdbfilerepprimaryrecovery.c' 'L145' 'FileRepPrimary_RunRecoveryInSync'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008568 CST,,,p59335,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'run consumer', mirroring role 'primary role' mirroring state 'sync' segment state 'initialization and recovery' filerep state 'initialization and recovery' process name(pid) 'primary consumer ack process(59335)' 'cdbfilerepprimaryack.c' 'L966' 'FileRepAckPrimary_RunConsumer'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008607 CST,,,p59335,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary consumer ack process(59335)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008642 CST,,,p59335,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary consumer ack process(59335)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.008682 CST,,,p59328,th-1729120192,"127.0.0.1","53902",2018-07-05 21:55:18 CST,0,,,seg-1,,,,,"WARNING","01000","PrimaryMirrorTransitionRequest (2) Result: Transition to primary/mirror mode PrimarySegment, data state InSync resulted in Error",,,,,,,0,,"primary_mirror_mode.c",1324,
2018-07-05 22:05:18.008717 CST,,,p59337,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","failure is detected in segment mirroring, failover requested",,,,,"mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' process name(pid) 'primary recovery process(59337)' filerep state 'initialization and recovery' ",,0,,"cdbfilerepprimary.c",265,
2018-07-05 22:05:18.028315 CST,,,p59331,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set segment state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary receiver ack process(59331)' 'cdbfilerep.c' 'L2457' 'FileRep_SetSegmentState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:18.028373 CST,,,p59331,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'set filerep state', mirroring role 'primary role' mirroring state 'sync' segment state 'in fault' filerep state 'fault' process name(pid) 'primary receiver ack process(59331)' 'cdbfilerepservice.c' 'L565' 'FileRepSubProcess_SetState'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:32.044313 CST,,,p59312,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","received immediate shutdown request",,,,,,,0,,"postmaster.c",4112,
2018-07-05 22:05:32.044937 CST,,,p59330,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary consumer ack process' process pid '59335' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(59330)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:32.044984 CST,,,p59330,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary recovery process' process pid '59337' exit status '2' ', mirroring role 'primary role'mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(59330)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:32.045051 CST,,,p59330,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary receiver ack process' process pid '59331' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(59330)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,
2018-07-05 22:05:32.045410 CST,,,p59330,th-1729120192,,,,0,,,seg-1,,,,,"LOG","00000","'process exit, process name 'primary sender process' process pid '59333' exit status '2' ', mirroring role 'primary role' mirroring state 'sync' segment state 'in immediate shutdown' filerep state 'not initialized' process name(pid) 'filerep main process(59330)' 'cdbfilerep.c' 'L2129' 'LogChildExit'",,,,,,,0,,"cdbfilerep.c",1839,

仔细读了一下日志，原因应该是从 filerep state ‘fault’ 这里开始导致的，并因此导致了 data state InSync resulted in Error 最后进程结束退出

拿着这个 filerep state ‘fault’ 搜一下呢？很欣喜的发现来到了无人区，全网只有一条结果 https://github.com/greenplum-db/gpdb/issues/1397 并且还没什么解决方案

于是接下来就开始进入自主探索阶段，首先想到去 QE_Primary_seg10_12 的 Mirror 节点看看，在 /data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Mirror_seg10_23/pg_log 我们从日志中发现这么一个提示

2018-07-05 13:54:14.508270 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""logging_collector"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.508881 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""log_destination"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.509455 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""log_directory"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.509982 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""log_destination"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.510511 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""log_directory"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.511090 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""logging_collector"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.511640 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""log_destination"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.512169 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""log_directory"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.512692 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""log_destination"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.513231 GMT,,,p40047,th63346752,,,,0,,,seg-1,,,,,"WARNING","01000","""log_directory"": setting is ignored because it is defunct",,,,,,,,"set_config_option","guc.c",4641,
2018-07-05 13:54:14.514707 GMT,,,p40050,th63346752,,,,0,,,seg-1,,,,,"FATAL","53100","could not write lock file ""postmaster.pid"": No space left on device",,,,,,,,"CreateLockFile","miscinit.c",1019,1 0x8c9846 postgres errstart + 0x1f6
2    0x8da406 postgres <symbol not found> + 0x8da406
3    0x768d3a postgres PostmasterMain + 0x88a
4    0x488ddb postgres main + 0x3bb
5    0x7f8a0281eb35 libc.so.6 __libc_start_main + 0xf5
6    0x488ef9 postgres <symbol not found> + 0x488ef9

说是磁盘没有空间了，顺手 df -h 看一眼

[gprds@10 /data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Mirror_seg10_23/pg_log]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        20G  3.5G   16G  19% /
devtmpfs         32G     0   32G   0% /dev
tmpfs            32G   16K   32G   1% /dev/shm
tmpfs            32G  3.2G   29G  11% /run
tmpfs            32G     0   32G   0% /sys/fs/cgroup
/dev/sda3        20G  857M   18G   5% /usr/local
/dev/sda4       1.8T   14G  1.7T   1% /data
/dev/md127p1    9.1T  8.6T     0 100% /data1
tmpfs           6.3G     0  6.3G   0% /run/user/0
tmpfs           6.3G     0  6.3G   0% /run/user/1000

我的天，9.1T 的盘被吃的干干净净，这是什么情况

[gprds@10 /data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10]$ du -h --max-depth=1
3.0T	./QE_Primary_seg7_9
78G	./QE_Mirror_seg4_17
5.5T	./QE_Primary_seg1_3
76G	./QE_Mirror_seg10_23
8.6T	.

进去这两个一看发现 core 得尸横遍野

连删都删不掉

[gprds@10 /data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Primary_seg1_3]$ rm core.61416
mkdir: cannot create directory '/data1/.Trash/gprds/2018070522': No space left on device

看起来是 alias 了 rm 命令，但是看了一下 alias 又没有

[gprds@10 /data1/gprds/RDSDIR/db_data/data/Greenplum/4.3.8.2/data/greenplum-da21lyj10/QE_Primary_seg1_3]$ alias
alias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'
alias grep='grep --color=auto'
alias l.='ls -d .* --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=auto'
alias vi='vim'
alias which='alias | /usr/bin/which --tty-only --read-alias --show-dot --show-tilde'

不管，先 echo "" > core.61416 清掉一个文件的内容再说，剩下的 core 一个个删除，但是发现删除的东西都被移动到 Trash 里面，空间并没有释放出来

于是尝试用 unlink 来删除 core 文件，发现这个是个好东西，可以把文件删掉，但是比较慢，删除一个 10G 的 core 文件要好一会，试了试 find ./ -name “core.48980” -delete 的方式，也是差不多一样慢，不过好在都可以把文件删掉

腾出来空间之后再次通过 gpstart -B 1 -t 3600 -v 来尝试启动集群，这回总算把集群启动起来了

但是这样并没有解决集群的数据恢复问题，于是冒着作死的精神，又把之前那份备份的 Mirror 节点的 datadir 拷贝过来了

居然可以起来，于是重新开始 gprecoverseg -F 的漫长过程

历史评论

house (2018-08-31 11:27:52):

牛逼哈哈

ZRJ (2018-08-31 19:18:23):

牛个毛线，然而最后并没有恢复成功

一顿操作猛如虎，一看战绩零比五

house (2018-09-01 00:14:11):

我一般就卡在google结果为空的步骤了