greenplum-db学习之安装篇
2017-03-25 21:59:54 
阿炯
本文记录了在Debian与CentOS下安装与维护Greenplum的相关过程，这是作者使用的编译选项：
./configure --prefix=/srv/gpdb --with-libedit-preferred --with-perl --with-openssl --with-libxml --enable-thread-safety --disable-orca
对中文提示的支持到2022年2月还是没有支持，因此'--enable-nls=zh_CN'这个选项还是无效的。为其添加专用用户：
useradd -m -r -u 2345 -g daemon -s /bin/bash -c 'greenplum dbcluster user' gpdb
-------------------------------deb-------------------------------
这里是在Devuan4下安装Greenplum-6.18的相关记录
问题：缺少libreadline-dev
configure: error: readline library not found
If you have readline already installed, see config.log for details on the
failure. It is possible the compiler isn't looking in the proper directory.
Use --without-readline to disable readline support.
问题：缺少zlib1g-dev，知道新的版本支持zstd，顺带将libzstd-dev也安装了
checking for inflate in -lz... no
configure: error: zlib library not found
If you have zlib already installed, see config.log for details on the
failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.
问题：缺少libapr1-dev
checking for apr-1-config... no
configure: error: apr-1-config is required for gpfdist, unable to find binary
问题：缺少libevent-dev
checking for library containing apr_getopt_long... none required
checking for library containing event_add... no
configure: error: libevent is required for gpfdist
问题：缺少libyaml-dev libssl-dev
checking for library containing yaml_parser_initialize... no
configure: WARNING: libyaml is not found. disabling transformations for gpfdist.
checking for CRYPTO_new_ex_data in -lcrypto... no
configure: error: library 'crypto' is required for OpenSSL
问题：缺少libxml2-dev
checking for xmlSaveToBuffer in -lxml2... no
configure: error: library 'xml2' (version >= 2.6.23) is required for XML support
问题：缺少libcurl4-openssl-dev
checking for curl-config... no
*** The curl-config script could not be found. Make sure it is
*** in your path, and that curl is properly installed.
*** Or see http://curl.haxx.se/
configure: error: Library requirements (curl) not met.
问题：缺少libbz2-dev
checking for BZ2_bzDecompress in -lbz2... no
configure: error: library 'bz2' is required for bzip2 support
问题：缺少官方新增的优化器环境
checking for strnicmp in -lxerces-c... no
configure: error: library xerces-c is required to build with Pivotal Query Optimizer
官方提供的大数据优化器(GPORCA)，这里不需要，使用--disable-orca编译参数跳过。
问题：缺少libperl-dev
checking for libperl... no
configure: error: libperl library is required for Perl
在make的过程中还有报依赖工具(bison flex)的错
ERROR: `bison' is missing on your system. It is needed to create the
file `gram.c'. You can either get bison from a GNU mirror site
or download an official distribution of PostgreSQL, which contains
pre-packaged bison output.
一次性安装
apt-get install libreadline-dev zlib1g-dev libzstd-dev libapr1-dev libevent-dev libyaml-dev libssl-dev libxml2-dev libcurl4-openssl-dev libbz2-dev libperl-dev bison flex -y
在make install时有报python，还是安装好python吧。
/bin/sh: 5: python: not found
make[2]: *** [Makefile:80：pygresql] 错误 127
make[2]: 离开目录“/root/soft/gpdb6.18/gpMgmt/bin”
make[1]: *** [Makefile:7：install-bin-recurse] 错误 2
make[1]: 离开目录“/root/soft/gpdb6.18/gpMgmt”
make: *** [GNUmakefile:78：install] 错误 2
python3-dev，但报错中明显需要python2，真是麻烦。。。安装好python2的相关包后，可能还需要做一个名为python链接，指向python2.7。不仅master需要，所有的segment也需要安装python2。
另外一些缺省组件的代码不是自动编译的，需要单独进行编译。
-------------------------------rpm-------------------------------
在CentOS8用源码编译并安装Greenplum-6.19
问题：
checking whether g++ supports C++11 features with +std=c++0x... no
checking whether g++ supports C++11 features with -h std=c++0x... no
configure: error: *** A compiler with support for C++11 language features is required.
解决：yum install gcc-c++
问题：
configure: WARNING:
*** Without Bison you will not be able to build PostgreSQL from Git nor
*** change any of the parser definition files. You can obtain Bison from
*** a GNU mirror site. (If you are using the official distribution of
*** PostgreSQL then you do not need to worry about this, because the Bison
*** output is pre-generated.)
解决：yum install bison
问题：
configure: WARNING:
*** Without Flex you will not be able to build PostgreSQL from Git nor
*** change any of the scanner definition files. You can obtain Flex from
*** a GNU mirror site. (If you are using the official distribution of
*** PostgreSQL then you do not need to worry about this because the Flex
*** output is pre-generated.)
解决：yum install flex
问题：
checking for flags to link embedded Perl... Can't locate ExtUtils/Embed.pm in @INC (you may need to install the ExtUtils::Embed module) (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5).
BEGIN failed--compilation aborted.
no configure: error: could not determine flags for linking embedded Perl.
This probably means that ExtUtils::Embed or ExtUtils::MakeMaker is not
installed.
解决：yum install perl-ExtUtils-Embed.noarch
问题：
checking for library containing readline... no
configure: error: readline library not found
If you have readline already installed, see config.log for details on the
failure. It is possible the compiler isn't looking in the proper directory.
Use --without-readline to disable readline support.
解决：yum install readline-devel
问题：
checking for ZSTD_compressCCtx in -lzstd... no
configure: error: zstd library not found
If you have libzstd already installed, see config.log for details on the
failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zstd to disable zstd support.
解决：yum install libzstd libzstd-devel
问题：
checking for library containing gss_init_sec_context... no
configure: error: could not find function 'gss_init_sec_context' required for GSSAPI
解决：yum install krb5-devel
问题：
checking for apr-1-config... no
configure: error: apr-1-config is required for gpfdist, unable to find binary
解决：yum install apr-devel
问题：
checking for library containing event_add... no
configure: error: libevent is required for gpfdist
解决：yum install libevent-devel
问题：
checking for xmlSaveToBuffer in -lxml2... no
configure: error: library 'xml2' (version >= 2.6.23) is required for XML support
解决：yum install libxml2-devel
问题：
checking for curl-config... no
*** The curl-config script could not be found. Make sure it is
*** in your path, and that curl is properly installed.
*** Or see http://curl.haxx.se/
configure: error: Library requirements (curl) not met.
解决：yum install libcurl-devel
问题：
checking for BZ2_bzDecompress in -lbz2... no
configure: error: library 'bz2' is required for bzip2 support
解决：yum install bzip2-devel
问题：
checking for strnicmp in -lxerces-c... no
configure: error: library xerces-c is required to build with Pivotal Query Optimizer
解决：yum install xerces-c-devel
make部分问题
问题：
In file included from include/reader.h:4,
from include/gpreader.h:4,
from src/gpcloud.cpp:38:
include/s3common_headers.h:8:10: 致命错误：openssl/hmac.h：No such file or directory
#include <openssl/hmac.h>
解决：yum install openssl-devel
如果安装完依赖库后，make失败，可以尝试make clean后再进行make。其安装过程与《PostgreSQL源码安装及配置》中相近。
--------------------------------------------------------------
需要生成或创立三个配置文件
--env.sh--
source /srv/gpdb/greenplum_path.sh
export PGPORT=5432
export MASTER_DATA_DIRECTORY=/home/gpdb/data/master/gpseg
--hostfile--
devf1
devf2
devf3
同样需要在操作系统的/etc/hosts中加主机相关的映射关系：
192.168.8.78 devfp
192.168.8.71 devf1
192.168.8.72 devf2
192.168.8.73 devf3
--gpinitsystem_config--
ARRAY_NAME="FreeOA GPDB"
SEG_PREFIX=gpseg
PORT_BASE=7000
# 根据需要修改下面的路径和主机名
declare -a DATA_DIRECTORY=(/home/gpdb/data /home/gpdb/data /home/gpdb/data)
MASTER_HOSTNAME=devfp
MASTER_DIRECTORY=/home/gpdb/data/master/gpseg
MASTER_PORT=5432
IP_ALLOW=0.0.0.0/0
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=3
ENCODING=UNICODE
MACHINE_LIST_FILE=./hostfile
简单说明下
ARRAY_NAME：设置阵列名称，默认Greenplum Data Platform。
SEG_PREFIX：设置segment的前缀，默认gpseg。
PORT_BASE：设置segment的起始端口，会从此端口往上增加，默认从40000开始。
DATA_DIRECTORY：设置segment primary的数据存储目录，有几个segment节点就需要设置几个数据存储目录。
MASTER_HOSTNAME：设置master的主机名。
MASTER_DIRECTORY：设置master的存储目录。
MASTER_PORT：设置master的端口，默认5432。
TRUSTED_SHELL：设置节点之间的信任方式，默认SSH。
CHECK_POINT_SEGMENTS：预写日志文件（WAL）数量，默认为8，这意味着为主机上的每个Segment或Master实例分配1088MB的WAL空间。
ENCODING=UNICODE：设置初始字符集，默认UNICODE（UTF-8）。
MACHINE_LIST_FILE：仅包含segment主机地址文件列表。
注意，其中所有需要的目录都是在创建数据存储区域时做好的。
尤其需要注意的是 MACHINE_LIST_FILE，指的是仅包含segment主机名，每行一个，主机名不要有空格。。
初始化 GPDB 集群
$ source env.sh
$ gpinitsystem -c gpinitsystem_config --locale=zh_CN.utf8 -a
注意：这里是非常简化过程，没有涉及standby与mirror这两部分。如果不出错就成功了，服务会自动起来，以后可以使用 gpstart 和 gpstop 起停服务。
初始化成功后，运行一下命令验证系统状态：
$ psql -l
$ gpstate
简单测试
--创建库
$ createdb test
--连接库
$ psql test
--使用generate_series生成表数据
test# CREATE TABLE freeoatab AS SELECT * FROM generate_series(1, 100000);
--统计每个节点的数量分布
test# SELECT gp_segment_id, count(1) FROM freeoatab GROUP BY gp_segment_id;
如果安装了新版本的优化器，可以查看其信息：
>select gp_opt_version();
gp_opt_version
---------------------------------------
Server has been compiled without ORCA
--------------------------------------------------------------
环境相关配置(操作系统)
数据库的调优所需要的知识比较广泛且有一定的深度，在这里也只是简单的提及，其后的原理就需要深入的学习了。
建议关闭SELINUX，并调整好IPTABLES中规则，下面的设置完成后，非常有必要重启该主机。分别涉及：
1.操作系统内核级参数优化(内核参数、文件、进线程数量限制、网络栈等)
2.硬盘IO优化(文件系统格式、配置参数、调度算法等)
3.时间源
4.SSH-Auto-Login
在Linux平台，推荐使用XFS文件系统，GP建议使用下面的挂载参数：rw,noatime,inode64,allocsize=16m
如在挂载XFS格式的设备/dev/sdb到目录/data，/etc/fstab中的配置如下：
/dev/sdb /data xfs rw,noatime,inode64,allocsize=16m 1 1
Linux磁盘I/O调度器对磁盘的访问支持不同的策略，默认的为CFQ，GP建议设置为deadline
要查看某驱动器的I/O调度策略，可通过如下命令查看，下面示例的为正确的配置：
# cat /sys/block/{devname}/queue/scheduler
noop anticipatory [deadline] cfq
修改磁盘I/O调度策略的方法为，修改/boot/grub/menu.lst文件的启动参数，在kernel一行的最后追加"elevator=deadline"
磁盘设备文件需要配置read-ahead(blockdev)值为65536
官方文档的推荐值为16384，但设定为65536更合理，该值设置的是预读扇区数，实际上预读的字节数是blockdev设置除以2，而GP缺省的blocksize为32KB，刚好与65536(32768B/32KB)对应。
检查某块磁盘的read-ahead设置：
# blockdev --getra devname
例如：
# blockdev --getra /dev/sda
65536
修改系统的read-ahead设置，可通过/etc/rc.d/rc.local来修改，在文件尾部追加如下代码：
# blockdev --setra 65536 /dev/mapper/vg00-LV_01
如需临时修改read-ahead设置，可通过执行下面的命令来实现：
# blockdev --setra bytes devname
例如：
# blockdev --setra 65536 /dev/sda
GP建议使用NTP(网络时间协议)来同步GPDB集群中所有主机的系统时钟。
在Segment 主机上，NTP应该配置Master 主机作为主时间源，而Standby作为备选时间源。在Master和Standby上配置NTP到首选的时间源(如果没有更好的选择可以选择Master自身作为最上端的事件源)。但要保证这组集群的时间是一致的，不能有过大的误差。软件可以使用ntpd或较新的chronyd。
--/etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072
* soft core unlimited
--/etc/sysctl.conf
kernel.shmmax = 8589934592
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.sem = 250 5120000 100 20480
kernel.sysrq = 1
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.msgmni = 2048
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.all.arp_filter = 1
net.ipv4.ip_local_port_range = 1025 65000
net.core.netdev_max_backlog = 10000
vm.overcommit_memory = 2
--------------------------------------------------------------
登录进入数据库数据中进行测试
CREATE TABLE t1 AS SELECT * FROM generate_series(1,10000000);
SELECT gp_segment_id, count(1) FROM t1 GROUP BY gp_segment_id;
create table fdfstat(
fid int4,
fsizebytes int8
)with (fillfactor=100, appendonly=true, compresstype=ZSTD, compresslevel=6) DISTRIBUTED BY ("fid");
COPY fdfstat FROM '/tmp/fdfs.149.35.220210.mist' DELIMITER ':';
GP集群信息查看
用gpstat来查看当前的状态，此命令也是排错时的必备工具：
gpstate -e #查看mirror的状态，这次的案例没有安装mirror
gpstate -f #查看standby master的状态
gpstate -s #查看整个GP群集的状态
gpstate -i #查看GP的版本
gpstate --help #帮助文档，可以查看gpstate更多用法。
性能测试
网络性能
网络测试选项包括：并行测试(-r N)、串行测试(-r n)、矩阵测试(-r M)。测试时运行一个网络测试程序从当前主机向远程主机传输5秒钟的数据流。缺省时，数据并行传输到每个远程主机，报告出传输的最小、最大、平均和中值速率，单位为MB/S。如果主体的传输速率低于预期(小于100MB/S)，可以使用-r n参数运行串行的网络测试以得到每个主机的结果。要运行矩阵测试，指定-r M参数，使得每个主机发送接收指定的所有其他主机的数据，这个测试可以验证网络层能否承受全矩阵工作负载。
gpcheckperf -f hostfile -r N -d /tmp > subnet1.out
检查磁盘IO、内存带宽
gpcheckperf -f hostfile -d /data/mirror -r ds
====================
== RESULT 2022-02-17T14:25:35.317752
====================
disk write avg time (sec): 14.24
disk write tot bytes: 50450726912
disk write tot bandwidth (MB/s): 3386.89
disk write min bandwidth (MB/s): 1060.70 [devf1]
disk write max bandwidth (MB/s): 1195.96 [devf2]
disk read avg time (sec): 13.66
disk read tot bytes: 50450726912
disk read tot bandwidth (MB/s): 3569.74
disk read min bandwidth (MB/s): 1003.62 [devf2]
disk read max bandwidth (MB/s): 1289.22 [devf3]
stream tot bandwidth (MB/s): 29537.70
stream min bandwidth (MB/s): 9406.80 [devf2]
stream max bandwidth (MB/s): 10468.40 [devf1]
检查每台Segment上的日期时间
gpssh -f hostfile -v date
关于gporca
gporca是greenplum新一代的优化器，在性能上有很大提升，在gporca即为旧式的优化(legacy)。
legacy是早期针对单节点 greenplum 而构建的，主要应用于OLTP场景，greenplum在不安装orca的情况下使用的是legacy优化器。
为什么要有两个优化器，其实都是有历史原因的，现在的greenplum使用MPP，主要应用场景变为OLAP场景；legacy对此虽然进行了修改，但是从架构设计上，使得其维护和添加新的功能越来越困难，所以有了GPORCA优化器。gporca 需要在每个节点都需要安装。
下面来列出另外一些不常见的操作问题，部分内容可能有重复。
---------------------------------------------------------------
PostgreSQL|Greenplum 启动或初始kernel SHMMAX parameter出错
原值为：33554432(32M)
可行的推荐值
1073741824(1G)
sysctl -w kernel.shmmax=1073741824
立即生效
sysctl -p
Then try to start postgresql
加到 8 GB:
$ sysctl -w kernel.shmmax=8589934592
$ sysctl -w kernel.shmall=2097152
一直有效的话，加入到 /etc/sysctl.conf 配置文件中：
kernel.shmmax = 1073741824
It occurs when either the system limit for the maximum number of semaphore sets (SEMMNI), or the system wide maximum number of semaphores (SEMMNS), would be exceeded. You need to raise the respective kernel parameter. Alternatively, reduce PostgreSQL's consumption of semaphores by reducing its max_connections parameter (currently 753).
The PostgreSQL documentation contains more information about configuring your system for PostgreSQL.",,,,,,"InternalIpcSemaphoreCreate","pg_sema.c",129,1 0x8c877c postgres errstart (elog.c:521)
sysctl -w kernel.sem="500 64000 50 150"
/etc/sysctl.conf
Add the following line:
fs.file-max=70000
在系统启动时修改semaphores空间参数
/sbin/sysctl -w kernel.sem="32000 1024000000 500 32000"
查看：
cat /proc/sys/kernel/sem
GPDB startup/initilization failed with "could not create semaphores: No space left on device"
Cause
As indicated by the error , the issue is due to improper kernel settings for semaphore and here there are not enough semaphores available at OS level.
Please NOTE: This error does not mean that you have run out of disk space.
Solution
Please refer Greenplum Database Installation Guide for recommended OS kernel parameters.
If the values are default as recommended on the Installation Guide , please verify if the values are set in double quotes, kernel will consider default values when used with quotes.
Current setting:
$ cat /etc/sysctl.conf |grep sem
kernel.sem="250 512000 100 2048"
Unable to Start PostgreSQL. kernel's SHMMAX parameter exceeded you limit
For Linux (example from the manual, to allow 16 GB):
$ sysctl -w kernel.shmmax=17179869184
$ sysctl -w kernel.shmall=4194304
---------------------------------------------------------------
所以在开始前，最好调整一下这个系统参数，不然会碰到下面的一系列问题。
env.sh
source /home/gpdb/greenplum_path.sh
export PGPORT=15432
export MASTER_DATA_DIRECTORY=/home/gpdb/data/master/gpseg1
hostfile
xd0
xd1
xd3
gpinitsystem_config
一个简单的配置文件如下：
ARRAY_NAME="FreeOA GPDB"
SEG_PREFIX=gpseg
PORT_BASE=40000
# 根据需要，修改下面的路径和主机名
declare -a DATA_DIRECTORY=(/home/gpdb/data /home/gpdb/data /home/gpdb/data)
MASTER_HOSTNAME=htcom
MASTER_DIRECTORY=/home/gpdb/data/master
MASTER_PORT=15432
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=6
ENCODING=UNICODE
MACHINE_LIST_FILE=./hostfile
调用配置文件开始初始化(gpinitsystem_config)过程中遇到的一些问题：
20170124:16:46:06:025117 gpinitsystem:htcom:postgres-[INFO]:-Commencing local /home/gpdb/bin/initdb -E UNICODE -D /home/gpdb/data/master/gpseg-1 --locale=en_US.utf8 --max_connections=250 --shared_buffers=128000kB --is_filerep_mirrored=no --backend_output=/home/gpdb/data/master/gpseg-1.initdb
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale en_US.utf8.
The default text search configuration will be set to "english".
creating directory /home/gpdb/data/master/gpseg-1 ... ok
creating subdirectories ... ok
selecting default max_connections ... initdb: error 256 from: "/home/gpdb/bin/postgres" --boot -x0 -F -c max_connections=250 -c shared_buffers=4000 -c max_fsm_pages=200000 < "/dev/null" > "/home/gpdb/data/master/gpseg-1.initdb" 2>&1
initdb: removing data directory "/home/gpdb/data/master/gpseg-1"
2017-01-24 16:46:07.122248 CST,,,p31735,th-591046880,,,,0,,,seg-1,,,,,"FATAL","XX000","could not create shared memory segment: 无效的参数 (pg_shmem.c:183)","Failed system call was shmget(key=1, size=176041872, 03600).","This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX. To reduce the request size (currently 176041872 bytes), reduce PostgreSQL's shared_buffers parameter (currently 4000) and/or its max_connections parameter (currently 253).
If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
The PostgreSQL documentation contains more information about shared memory configuration.",,,,,,"InternalIpcMemoryCreate","pg_shmem.c",183,1 0x8c877c postgres errstart (elog.c:521)
2 0x75a718 postgres <symbol not found> (pg_shmem.c:145)
3 0x75a9d2 postgres PGSharedMemoryCreate (pg_shmem.c:387)
4 0x7af330 postgres CreateSharedMemoryAndSemaphores (ipci.c:241)
5 0x8d6969 postgres BaseInit (postinit.c:542)
6 0x54e324 postgres AuxiliaryProcessMain (bootstrap.c:415)
7 0x49c4d4 postgres main (main.c:192)
8 0x7f71db153ead libc.so.6 __libc_start_main + 0xfd
9 0x49c59d postgres <symbol not found> + 0x49c59d
20170124:16:46:07:025117 gpinitsystem:htcom:postgres-[INFO]:-Start Function ERROR_EXIT
postgres@htcom:/home/gpdb$ ./bin/gpinitsystem -c gpinitsystem_config -a
...
.......................
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:------------------------------------------------
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:-Parallel process exit status
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:------------------------------------------------
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:-Total processes marked as completed = 6
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:-Total processes marked as killed = 0
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[WARN]:-Total processes marked as failed = 3 <<<<<
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:------------------------------------------------
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[FATAL]:-Errors generated from parallel processes
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:-Dumped contents of status file to the log file
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:-Building composite backout file
20170124:16:53:30:gpinitsystem:htcom:postgres-[FATAL]:-Failures detected, see log file /home/postgresql/gpAdminLogs/gpinitsystem_20170124.log for more detail Script Exiting!
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[WARN]:-Script has left Greenplum Database in an incomplete state
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[WARN]:-Run command /bin/bash /home/postgresql/gpAdminLogs/backout_gpinitsystem_postgres_20170124_165242 to remove these changes
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:-Start Function BACKOUT_COMMAND
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:-End Function BACKOUT_COMMAND
20170124:17:31:14:032318 gpstart:htcom:postgres-[DEBUG]:-[worker3] finished cmd: remote segment starts on host 'xd3' cmdStr='ssh -o 'StrictHostKeyChecking no' xd3 ". /home/gpdb/greenplum_path.sh; $GPHOME/sbin/gpsegstart.py -C en_US.utf8:en_US.utf8:en_US.utf8 -M mirrorless -V 'postgres (Greenplum Database) 5.0.0 build dev' -n 9 --era 39a9ae0cf4490730_170124173112 -t 600 -v -D '8|6|p|p|s|u|xd3|xd3|40000|None|/home/gpdb/data/gpseg6||' -D '9|7|p|p|s|u|xd3|xd3|40001|None|/home/gpdb/data/gpseg7||' -D '10|8|p|p|s|u|xd3|xd3|40002|None|/home/gpdb/data/gpseg8||' -l '/home/postgresql/gpAdminLogs'"' had result: cmd had rc=1 completed=True halted=False
stdout=''
stderr='Error: unable to import module: No module named psutil
由于它的初始脚本是用python开发的，而这个工具脚本运行过程中需要一些第三方的模块，因此就必须先安装这些模块：
pip install psi lockfile paramiko setuptools epydoc psutil
在初始化greenplum时，报了一个错误
20161012:14:58:57:005531 gpcreateseg.sh:master:gpadmin-[INFO]:-Start Function ED_PG_CONF
20161012:14:58:57:005531 gpcreateseg.sh:master:gpadmin-[WARN]:-Failed to insert port=40000 in /data/gpadmin/gpdata/primary/gpseg2/postgresql.conf on sdw3
20161012:14:58:57:005531 gpcreateseg.sh:master:gpadmin-[INFO]:-End Function ED_PG_CONF
20161012:14:58:57:005531 gpcreateseg.sh:master:gpadmin-[FATAL][2]:-Failed Update port number to 40000
网上搜了一下，需要安装ed
yum install -y ed
官方文档没有提及这个依赖包，故此需要注意一下，而且每个节点包含segment都要把ed安装一遍
将程序文件从主节点上复制一份到所有的子节点：
rsync -av --exclude=master /home/gpdb/ xd3:/home/gpdb/
---------------------------------------------------------------
在家中的crux linux上的实验过程
crux上的openssh没有ssh-copy-id工具，需要手动操作一下：
ssh postgres@crux1 'mkdir -p ~/.ssh'
cat ~/.ssh/id_rsa.pub | ssh postgres@crux1 'cat >> ~/.ssh/authorized_keys'
安装过程：
postgres@pserv:/home/gpdb$ bin/gpinitsystem -c gpinit_system_config --max_connections=64 --shared_buffers=256MB
bin/lib/gp_bash_functions.sh: line 77: return: Problem in gp_bash_functions, command 'netstat' not found in COMMAND path. You will need to edit the script named gp_bash_functions.sh to properly locate the needed commands for your platform.: numeric argument required
20170225:20:02:16:000351 gpinitsystem:pserv:postgres-[INFO]:-Checking configuration parameters, please wait...
20170225:20:02:16:000351 gpinitsystem:pserv:postgres-[INFO]:-Reading Greenplum configuration file gpinit_system_config
20170225:20:02:16:000351 gpinitsystem:pserv:postgres-[INFO]:-Locale has not been set in gpinit_system_config, will set to default value
20170225:20:02:16:000351 gpinitsystem:pserv:postgres-[FATAL]:-Unable to locate locale value en_US.utf8 on this host
20170225:20:02:17:000351 gpinitsystem:pserv:postgres-[INFO]:-Select another locale value via -n option to this utility
20170225:20:02:17:000351 gpinitsystem:pserv:postgres-[INFO]:-Available locale values have been dumped to the log file
20170225:20:02:17:gpinitsystem:pserv:postgres-[FATAL]:-Unable to continue Script Exiting!
我机器上的locale有：
postgres@pserv:/home/gpdb$ locale -a
C
POSIX
zh_CN.utf8
手动生成一下en_US的locale
localedef -i en_US -f UTF-8 en_US.UTF-8
postgres@pserv:/home/gpdb$ bin/gpinitsystem -c gpinit_system_config --max_connections=64 --shared_buffers=256MB
bin/lib/gp_bash_functions.sh: line 77: return: Problem in gp_bash_functions, command 'netstat' not found in COMMAND path. You will need to edit the script named gp_bash_functions.sh to properly locate the needed commands for your platform.: numeric argument required
20170225:20:31:31:002137 gpinitsystem:pserv:postgres-[INFO]:-Checking configuration parameters, please wait...
20170225:20:31:31:002137 gpinitsystem:pserv:postgres-[INFO]:-Reading Greenplum configuration file gpinit_system_config
20170225:20:31:31:002137 gpinitsystem:pserv:postgres-[INFO]:-Locale has not been set in gpinit_system_config, will set to default value
20170225:20:31:31:002137 gpinitsystem:pserv:postgres-[INFO]:-Locale set to en_US.utf8
20170225:20:31:32:002137 gpinitsystem:pserv:postgres-[INFO]:-No DATABASE_NAME set, will exit following template1 updates
20170225:20:31:32:002137 gpinitsystem:pserv:postgres-[WARN]:-Master open file limit is 1024 should be >= 65535
20170225:20:31:32:002137 gpinitsystem:pserv:postgres-[INFO]:-Checking configuration parameters, Completed
20170225:20:31:32:002137 gpinitsystem:pserv:postgres-[INFO]:-Commencing multi-home checks, please wait...
...
20170225:20:31:34:002137 gpinitsystem:pserv:postgres-[INFO]:-Configuring build for standard array
20170225:20:31:34:002137 gpinitsystem:pserv:postgres-[INFO]:-Commencing multi-home checks, Completed
20170225:20:31:34:002137 gpinitsystem:pserv:postgres-[INFO]:-Building primary segment instance array, please wait...
.........
20170225:20:31:42:002137 gpinitsystem:pserv:postgres-[INFO]:-Building group mirror array type , please wait...
.........
20170225:20:31:50:002137 gpinitsystem:pserv:postgres-[INFO]:-Checking Master host
20170225:20:31:50:002137 gpinitsystem:pserv:postgres-[INFO]:-Checking new segment hosts, please wait...
20170225:20:31:51:002137 gpinitsystem:pserv:postgres-[WARN]:-Host pserv open files limit is 1024 should be >= 65535
20170225:20:31:52:gpinitsystem:pserv:postgres-[FATAL]:-No /home/gpdb/lib on segment instance crux1 Script Exiting!
crux下没有netstat指令了，取而代之的是ss，在shell的初始文件中做一个别名吧：
alias netstat='/sbin/ss'
将master机器上的程序文件复制到各个节点上
rsync -av --exclude=master /home/gpdb/ crux[1-3]:/home/gpdb/
Warning: Permanently added 'crux1,192.168.8.11' (ECDSA) to the list of known hosts.
Permission denied (publickey,password,keyboard-interactive).
[FATAL]:-Unexpected EOF on RemotePysync output stream
Permission denied (publickey,password,keyboard-interactive).
Permission denied (publickey,password,keyboard-interactive).
[FATAL]:-Unexpected EOF on RemotePysync output stream
[FATAL]:-Unexpected EOF on RemotePysync output stream
20170225:20:37:57:010898 gpcreateseg.sh:pserv:postgres-[FATAL]:- Command export GPHOME=/home/gpdb; . /home/gpdb/greenplum_path.sh; /home/gpdb/bin/lib/pysync.py -x pg_log -x postgresql.conf -
x postmaster.pid /home/gpdb/primary/gpseg7 \[crux1\]:/home/gpdb/mirror/gpseg7 on crux3 failed with error status 3
20170225:20:37:57:011228 gpcreateseg.sh:pserv:postgres-[FATAL]:- Command export GPHOME=/home/gpdb; . /home/gpdb/greenplum_path.sh; /home/gpdb/bin/lib/pysync.py -x pg_log -x postgresql.conf -
x postmaster.pid /home/gpdb/primary/gpseg8 \[crux1\]:/home/gpdb/mirror/gpseg8 on crux3 failed with error status 3
20170225:20:37:57:010590 gpcreateseg.sh:pserv:postgres-[FATAL]:- Command export GPHOME=/home/gpdb; . /home/gpdb/greenplum_path.sh; /home/gpdb/bin/lib/pysync.py -x pg_log -x postgresql.conf -
x postmaster.pid /home/gpdb/primary/gpseg6 \[crux1\]:/home/gpdb/mirror/gpseg6 on crux3 failed with error status 3
20170225:20:37:57:010898 gpcreateseg.sh:pserv:postgres-[INFO]:-End Function RUN_COMMAND_REMOTE
20170225:20:37:57:010590 gpcreateseg.sh:pserv:postgres-[INFO]:-End Function RUN_COMMAND_REMOTE
20170225:20:37:57:011228 gpcreateseg.sh:pserv:postgres-[INFO]:-End Function RUN_COMMAND_REMOTE
20170225:20:37:57:010898 gpcreateseg.sh:pserv:postgres-[FATAL][7]:-Failed remote copy of segment data directory from crux3 to crux1
20170225:20:37:57:011228 gpcreateseg.sh:pserv:postgres-[FATAL][8]:-Failed remote copy of segment data directory from crux3 to crux1
20170225:20:37:57:010590 gpcreateseg.sh:pserv:postgres-[FATAL][6]:-Failed remote copy of segment data directory from crux3 to crux1
20170225:20:37:58:008423 gpinitsystem:pserv:postgres-[INFO]:-End Function PARALLEL_WAIT
20170225:20:37:58:008423 gpinitsystem:pserv:postgres-[INFO]:-End Function PARALLEL_COUNT
20170225:20:37:58:008423 gpinitsystem:pserv:postgres-[INFO]:-Start Function PARALLEL_SUMMARY_STATUS_REPORT
20170225:20:37:58:008423 gpinitsystem:pserv:postgres-[INFO]:------------------------------------------------
20170225:20:37:58:008423 gpinitsystem:pserv:postgres-[INFO]:-Parallel process exit status
20170225:20:37:58:008423 gpinitsystem:pserv:postgres-[INFO]:------------------------------------------------
20170225:20:37:59:008423 gpinitsystem:pserv:postgres-[INFO]:-Total processes marked as completed = 0
20170225:20:37:59:008423 gpinitsystem:pserv:postgres-[INFO]:-Total processes marked as killed = 0
20170225:20:37:59:008423 gpinitsystem:pserv:postgres-[WARN]:-Total processes marked as failed = 9 <<<<<
20170225:20:37:59:008423 gpinitsystem:pserv:postgres-[INFO]:------------------------------------------------
20170225:20:37:59:008423 gpinitsystem:pserv:postgres-[INFO]:-End Function PARALLEL_SUMMARY_STATUS_REPORT
FAILED:crux2~50002~/home/gpdb/mirror/gpseg2~13~2~51002
FAILED:crux2~50000~/home/gpdb/mirror/gpseg0~11~0~51000
FAILED:crux2~50001~/home/gpdb/mirror/gpseg1~12~1~51001
FAILED:crux3~50001~/home/gpdb/mirror/gpseg4~15~4~51001
FAILED:crux3~50000~/home/gpdb/mirror/gpseg3~14~3~51000
FAILED:crux3~50002~/home/gpdb/mirror/gpseg5~16~5~51002
FAILED:crux1~50001~/home/gpdb/mirror/gpseg7~18~7~51001
FAILED:crux1~50000~/home/gpdb/mirror/gpseg6~17~6~51000
FAILED:crux1~50002~/home/gpdb/mirror/gpseg8~19~8~51002
20170225:20:37:59:008423 gpinitsystem:pserv:postgres-[INFO]:-End Function CREATE_QES_MIRROR
看来需要将各个segment之间都要做成ssh-autologin
在集群完成创建后，开始破坏性的测试：在insert|update时，将其中的一台segment主机reboot。
test=# insert into wnb select * from wnqw;
NOTICE: Releasing segworker groups to finish aborting the transaction.
ERROR: FTS detected connection lost during dispatch to seg0 192.168.8.11:40000 pid=3610:
ERROR: Command could not be dispatch to segment seg0 192.168.8.11:40000 pid=3610: another command is already in progress
时间：154611.104 ms
这期间将一台seg重启了，下面也是如此：
test=# update wnqw set status=2 where id between 109491419 and 110491418;
WARNING: mirror failure, could not complete mirrored request identifier 'base/16384/16420.1' ack state 'waiting for ack', failover requested (seg5 192.168.8.12:40002 pid=7127)
提示: run gprecoverseg to re-establish mirror connectivity
WARNING: mirror failure, could not complete operation on mirror, failover requested (seg5 192.168.8.12:40002 pid=7127)
描述: identifier 'base/16384/16420.1' operation 'not specified' relation type 'append only' message count '-1'
提示: run gprecoverseg to re-establish mirror connectivity
NOTICE: Releasing segworker groups to finish aborting the transaction.
ERROR: FTS detected connection lost during dispatch to seg6 192.168.8.13:40000 pid=6149:
ERROR: Command could not be dispatch to segment seg6 192.168.8.13:40000 pid=6149: another command is already in progress
时间：75945.877 ms
postgres@pserv:/home/gpdb$ bin/gpstate -m
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:-Starting gpstate with args: -m
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.0.0 build dev'
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.0.0 build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (CRUX-x86_64-multilib) 5.2.0 compiled on Feb 2 2017 18:39:11'
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:-Obtaining Segment details from master...
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:--Current GPDB mirror list and status
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:--Type = Group
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- Mirror Datadir Port Status Data Status
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg0 50000 Passive Synchronized
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg1 50001 Passive Synchronized
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg2 50002 Passive Synchronized
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg3 50000 Passive Synchronized
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg4 50001 Passive Synchronized
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg5 50002 Passive Synchronized
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg6 50000 Passive Resynchronizing
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg7 50001 Passive Resynchronizing
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg8 50002 Passive Resynchronizing
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
修改成功后，再选一台seg主机再reboot。
test=# update wnqw set update_time=now() where id>109491418;
WARNING: mirror failure, could not queue message to be mirrored, failover requested (seg0 192.168.8.11:40000 pid=4198)
描述: identifier 'base/16384/16420.1' operation 'write' relation type 'append only' message count '-1'
提示: run gprecoverseg to re-establish mirror connectivity
NOTICE: Releasing segworker groups to finish aborting the transaction.
ERROR: FTS detected connection lost during dispatch to seg3 192.168.8.12:40000 pid=8560:
ERROR: Command could not be dispatch to segment seg3 192.168.8.12:40000 pid=8560: another command is already in progress
时间：800617.362 ms
test=# update wnb set update_time=now() where id>110091418;
NOTICE: Releasing segworker groups to finish aborting the transaction.
ERROR: FTS detected connection lost during dispatch to seg0 192.168.8.11:40000 pid=5997:
ERROR: Command could not be dispatch to segment seg0 192.168.8.11:40000 pid=5997: another command is already in progress
时间：114056.240 ms
不做恢复再更新一次
test=# update wnb set update_time=now() where id>110091418;
UPDATE 3400024
时间：177193.271 ms
test=# select min(update_time),max(update_time) from wnb where id>110091418;
min | max
----------------------------+----------------------------
2017-02-26 18:41:34.061123 | 2017-02-26 18:41:34.061123
(1 行记录)
时间：27466.814 ms
postgres@pserv:/home/gpdb$ bin/gpstate -m
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:-Starting gpstate with args: -m
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.0.0 build dev'
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.0.0 build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (CRUX-x86_64-multilib) 5.2.0 compiled on Feb 2 2017 18:39:11'
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:-Obtaining Segment details from master...
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:--Current GPDB mirror list and status
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:--Type = Group
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- Mirror Datadir Port Status Data Status
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg0 50000 Acting as Primary Resynchronizing
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg1 50001 Acting as Primary Resynchronizing
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg2 50002 Acting as Primary Resynchronizing
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg3 50000 Passive Synchronized
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg4 50001 Passive Synchronized
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg5 50002 Passive Synchronized
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg6 50000 Passive Resynchronizing
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg7 50001 Passive Resynchronizing
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg8 50002 Passive Synchronized
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
20170226:18:52:23:011124 gpstate:pserv:postgres-[WARNING]:-3 segment(s) configured as mirror(s) are acting as primaries
postgres@pserv:/home/gpdb$ bin/gpstate -m
20170226:19:48:12:013555 gpstate:pserv:postgres-[INFO]:-Starting gpstate with args: -m
20170226:19:48:12:013555 gpstate:pserv:postgres-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.0.0 build dev'
20170226:19:48:12:013555 gpstate:pserv:postgres-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.0.0 build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (CRUX-x86_64-multilib) 5.2.0 compiled on Feb 2 2017 18:39:11'
20170226:19:48:12:013555 gpstate:pserv:postgres-[INFO]:-Obtaining Segment details from master...
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:--Current GPDB mirror list and status
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:--Type = Group
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- Mirror Datadir Port Status Data Status
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg0 50000 Passive Synchronized
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg1 50001 Passive Synchronized
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg2 50002 Acting as Primary Change Tracking
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg3 50000 Passive Resynchronizing
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg4 50001 Passive Resynchronizing
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg5 50002 Passive Resynchronizing
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg6 50000 Acting as Primary Resynchronizing
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg7 50001 Acting as Primary Resynchronizing
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg8 50002 Acting as Primary Resynchronizing
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
20170226:19:48:13:013555 gpstate:pserv:postgres-[WARNING]:-4 segment(s) configured as mirror(s) are acting as primaries
20170226:19:48:13:013555 gpstate:pserv:postgres-[WARNING]:-1 mirror segment(s) acting as primaries are in change tracking
postgres@pserv:/home/gpdb$ bin/gpstate
20170226:19:48:41:013617 gpstate:pserv:postgres-[INFO]:-Starting gpstate with args:
20170226:19:48:41:013617 gpstate:pserv:postgres-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.0.0 build dev'
20170226:19:48:41:013617 gpstate:pserv:postgres-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.0.0 build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (CRUX-x86_64-multilib) 5.2.0 compiled on Feb 2 2017 18:39:11'
20170226:19:48:41:013617 gpstate:pserv:postgres-[INFO]:-Obtaining Segment details from master...
20170226:19:48:41:013617 gpstate:pserv:postgres-[INFO]:-Gathering data from segments...
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:-Greenplum instance status summary
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:-----------------------------------------------------
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Master instance = Active
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Master standby = No master standby configured
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total segment instance count from metadata = 18
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:-----------------------------------------------------
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Primary Segment Status
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:-----------------------------------------------------
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total primary segments = 9
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total primary segment valid (at master) = 8
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total primary segment failures (at master) = 1 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number of postmaster.pid files missing = 9 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number of postmaster.pid files found = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number of postmaster.pid PIDs missing = 9 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number of postmaster.pid PIDs found = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number of /tmp lock files missing = 9 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number of /tmp lock files found = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number postmaster processes missing = 9 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number postmaster processes found = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:-----------------------------------------------------
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Mirror Segment Status
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:-----------------------------------------------------
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total mirror segments = 9
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total mirror segment valid (at master) = 9
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total mirror segment failures (at master) = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number of postmaster.pid files missing = 9 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number of postmaster.pid files found = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number of postmaster.pid PIDs missing = 9 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number of postmaster.pid PIDs found = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number of /tmp lock files missing = 9 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number of /tmp lock files found = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number postmaster processes missing = 9 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number postmaster processes found = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number mirror segments acting as primary segments = 4 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number mirror segments acting as mirror segments = 5
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:-----------------------------------------------------
postgres@htcom:/home/gpdb$ bin/gprecoverseg
postgres@htcom:/home/gpdb$ bin/gpstate -s
postgres@htcom:/home/gpdb$ bin/gpstate -m
postgres@htcom:/home/gpdb$ bin/gprecoverseg -r
20170227:16:43:53:008719 gprecoverseg:htcom:postgres-[INFO]:-Starting gprecoverseg with args: -r
20170227:16:43:53:008719 gprecoverseg:htcom:postgres-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.0.0 build dev'
20170227:16:43:53:008719 gprecoverseg:htcom:postgres-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.0.0 build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (Debian 4.7.2-5) 4.7.2 compiled on Jan 24 2017 16:17:20'
20170227:16:43:54:008719 gprecoverseg:htcom:postgres-[INFO]:-Checking if segments are ready to connect
20170227:16:43:54:008719 gprecoverseg:htcom:postgres-[INFO]:-Obtaining Segment details from master...
20170227:16:43:54:008719 gprecoverseg:htcom:postgres-[INFO]:-Obtaining Segment details from master...
20170227:16:43:54:008719 gprecoverseg:htcom:postgres-[CRITICAL]:-gprecoverseg failed. (Reason='Down segments still exist. All segments must be up to rebalance.') exiting...
不是每次都能恢复成功，有时要执行gpstop，后又gpstart。主节点大都能起来，数据的安全性还是有保障的。
-------------------------------
20170323:15:46:10:001297 gpinitsystem:htcom:postgres-[WARN]:-Host htcom open files limit is 1024 should be >= 65535
这个文件最大打开数不只是调整系统的，还要调对应用户的。
The database cluster will be initialized with locale en_US.utf8.
The default text search configuration will be set to "english".
creating directory /home/gpdb/master/gpseg-1 ... ok
creating subdirectories ... ok
selecting default max_connections ... initdb: error 256 from: "/home/gpdb/bin/postgres" --boot -x0 -F -c max_connections=250 -c shared_buffers=4000 -c max_fsm_pages=200000 < "/dev/null" > "/home/gpdb/master/gpseg-1.initdb" 2>&1
initdb: removing data directory "/home/gpdb/master/gpseg-1"
2017-03-23 15:54:03.548500 CST,,,p8061,th250627872,,,,0,,,seg-1,,,,,"FATAL","XX000","could not create shared memory segment: 无效的参数 (pg_shmem.c:183)","Failed system call was shmget(key=1, size=176041872, 03600).","This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX. To reduce the request size (currently 176041872 bytes), reduce PostgreSQL's shared_buffers parameter (currently 4000) and/or its max_connections parameter (currently 253).
If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
The PostgreSQL documentation contains more information about shared memory configuration.",,,,,,"InternalIpcMemoryCreate","pg_shmem.c",183,1 0x8c877c postgres errstart (elog.c:521)
2 0x75a718 postgres <symbol not found> (pg_shmem.c:145)
3 0x75a9d2 postgres PGSharedMemoryCreate (pg_shmem.c:387)
4 0x7af330 postgres CreateSharedMemoryAndSemaphores (ipci.c:241)
5 0x8d6969 postgres BaseInit (postinit.c:542)
6 0x54e324 postgres AuxiliaryProcessMain (bootstrap.c:415)
7 0x49c4d4 postgres main (main.c:192)
8 0x7f4c0d402ead libc.so.6 __libc_start_main + 0xfd
9 0x49c59d postgres <symbol not found> + 0x49c59d
20170323:15:54:03:003438 gpinitsystem:htcom:postgres-[INFO]:-Start Function ERROR_EXIT
20170323:15:54:03:gpinitsystem:htcom:postgres-[FATAL]:- Command /home/gpdb/bin/initdb -E UNICODE -D /home/gpdb/master/gpseg-1 --locale=en_US.utf8 --max_connections=250 --shared_buffers=128000kB --is_filerep_mirrored=no --backend_output=/home/gpdb/master/gpseg-1.initdb failed with error status 1, see log file /home/postgres/gpAdminLogs/gpinitsystem_20170323.log for more detail Script Exiting!
其中的一个节点不能初始成功：
...
creating template1 database in /mnt/pc/gpdb/gpseg1/base/1 ... 2017-03-23 16:01:18.635500 CST,,,p29513,th-1561041120,,,,0,,,seg-1,,,,,"FATAL","XX000","could not create semaphores:
设备上没有空间 (pg_sema.c:129)","Failed system call was semget(111, 17, 03600).","This error does *not* mean that you have run out of disk space.
It occurs when either the system limit for the maximum number of semaphore sets (SEMMNI), or the system wide maximum number of semaphores (SEMMNS), would be exceeded. You need to raise the respective kernel parameter. Alternatively, reduce PostgreSQL's consumption of semaphores by reducing its max_connections parameter (currently 753).
The PostgreSQL documentation contains more information about configuring your system for PostgreSQL.",,,,,,"InternalIpcSemaphoreCreate","pg_sema.c",129,1 0x8c877c postgres errst
art (elog.c:521)
2 0x75a1b9 postgres PGSemaphoreCreate (pg_sema.c:114)
3 0x7b9df1 postgres InitProcGlobal (proc.c:216)
4 0x7af5bc postgres CreateSharedMemoryAndSemaphores (ipci.c:340)
5 0x8d6969 postgres BaseInit (postinit.c:542)
6 0x54e324 postgres AuxiliaryProcessMain (bootstrap.c:415)
7 0x49c4d4 postgres main (main.c:192)
8 0x7f69a1444ead libc.so.6 __libc_start_main + 0xfd
9 0x49c59d postgres <symbol not found> + 0x49c59d
child process exited with exit code 1
initdb: removing data directory "/mnt/pc/gpdb/gpseg1"
20170323:16:01:18:028704 gpcreateseg.sh:htcom:postgres-[INFO]:-Start Function BACKOUT_COMMAND
20170323:16:01:18:028704 gpcreateseg.sh:htcom:postgres-[INFO]:-End Function BACKOUT_COMMAND
20170323:16:01:18:028704 gpcreateseg.sh:htcom:postgres-[INFO]:-Start Function BACKOUT_COMMAND
20170323:16:01:18:028704 gpcreateseg.sh:htcom:postgres-[INFO]:-End Function BACKOUT_COMMAND
20170323:16:01:18:028704 gpcreateseg.sh:htcom:postgres-[FATAL][1]:-Failed to start segment instance database htcom /mnt/pc/gpdb/gpseg1
ok
loading file-system persistent tables for template1 (mirrored = false) ...
ok
loading file-system persistent tables for template1 (mirrored = false) ...
ok
...
还是老问题，要调整内核参数(crux上的这个参数设置的比较合理，基本不用做调整，下面这个就是crux上的参数值)：
/sbin/sysctl -w kernel.sem="32000 1024000000 500 32000"
创建支持压缩表的方法
CREATE TABLE "wnm2qz" (
...
)
WITH (appendonly=true, compresstype=QUICKLZ, compresslevel=1)
DISTRIBUTED BY ("id");
WITH (appendonly=true, compresstype=QUICKLZ, compresslevel=7)
[Err] ERROR: compresslevel=7 is out of range for quicklz (should be 1)
才发现系统中根本没有其链接库
ERROR: 22000: quicklz compression not supported (quicklz_compression.c:26) (seg2 192.168.0.9:40002 pid=21991)
位置: cdbdisp_finishCommand, cdbdisp.c:239
CREATE TABLE "wnm2z" (
...
)
WITH (appendonly=true, compresstype=ZLIB, compresslevel=6)
DISTRIBUTED BY ("id");
-------------------------------
个人的使用感受
本人是一名运维人员，只是想测试一下'大量'数据的情况下查询的效率。个人认为它很优秀的地方有：
不错的在线压缩功能
按特定条件的分布存放，分区管理
充分利用它所在的每台机器的资源
数据的安全性有保障的且恢复不难
一些不错的功能点：
单一入口
支持列存储
个人更喜欢在单机上玩伪分布式的方式，将机器的存储方式改为多组阵列1(raid1)。
mkdir -p /home/gpdb/primary /home/gpdb/mirror && chown -R postgres.daemon /home/gpdb
其配置文件也更简单，下面就是我的这种方式的配置文件：
ARRAY_NAME="FreeOA GPDB"
SEG_PREFIX=gpseg
PORT_BASE=40000
declare -a DATA_DIRECTORY=(/mnt/pb/gpdb /mnt/pc/gpdb /mnt/pd/gpdb)
MASTER_PORT=15432
MASTER_HOSTNAME=htcom
MASTER_DIRECTORY=/home/gpdb/master
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=8
ENCODING=UNICODE
MACHINE_LIST_FILE=./hostfile
hostfile也只需写本机名称即可
hto@htcom:/home/gpdb$ more hostfile
htcom
greenplum的备份，提供了gp_dump做并行备份，master和segment节点同时执行备份操作，另外gp_crondump会定期执行备份操作；支持PostgreSQL的pg_dump和pg_dumpall命令，但是其将所有的数据写成一个文件保存在master上，使用受限。greenplum数据库的启动gpstart，关闭gpstop，重启gpstop -r，强制关闭gpstop -f ，修改参数后不重启DB加载参数gpstop -u ，仅启动mastergpstart -m；
系统架构上一些理解
冗余和镜像：
基于PostgreSQL 8.2.13，内部修改以支持并行结构，segments上保存数据，节点间内部通信默认采用UDP协议，用额外的packet负责控制信息，代替TCP协议且取得不错的效果。
segment镜像和segment一定保存在不同的host上，master如果连不上segment，就把它标记为invalid状态，如果下次连上了，就重置为valid状态。如果系统没配置镜像，那么master在检测到invalid的segment时就会关闭数据库，来保证数据不出错；如果系统配置了镜像，那么系统在read-only模式和continue模式下的处理方式是不同的，前者不允许DDL和DML操作，可在线恢复，后者的操作必须限制在非invalid的segment上的数据，而且invalid的segment恢复时必须重启数据库系统。
master镜像就是master的在线standby，因为master上面只保存目录表和系统日志，因此在standby上面有一个事务重复记录进程(gpsyncagent)连接master以同步，一旦master连不上即坏掉，该进程停止工作，并让standby从master上一个成功事务操作之后开始接管系统(PS:master出故障前，standby上面只有gpsyncagent一个进程运行)。互连冗余，意指每个segment至少双网卡，一用网络连接，二用内部数据传输，提高通信可靠性。(详细参考白皮书的第五章系统管理的高可用性章节)
数据并行加载：
通过外部表(external table)，使用并行文件服务器(gpfdist)，可实现并行的数据加载。策略是M:N，即每个segment都会去连接每个gpfdist服务器。可通过postgresql.conf配置文件的gp_external_max_segs参数调整连接一个gpfdist服务器的segments个数，默认是64。
换句话说，external tables就是数据库将text文件按照列分割展示为数据库表的形式，可以执行select、join、sort这样的SQL操作了，相当于ETL的“抽取”部分。当然这个表是read-only的，不能执行DML操作，也不能建索引，并且数据展示依赖于gpfdist服务进程的存在。最后可以用CREATE TABLE X AS SELECT和INSERT INTO SELECT这样的语句实现ETL的“加载”部分，在加载的性能上，要考虑以下3点：先导入后建索引；导入后执行analyze更新query planner记录的信息；导入时遇到错误的话，最后要执行vacuum，用来释放失败的导入操作占用的空间。
管理和监视
简单说，管理包括从数据库安装到数据并行加载的很多命令，都保存在$GPHOME/bin目录下。要想对greenplum进行性能监视，就要在每个host上安装一个agent，一般每15秒采集一次数据，发送到master上。
分布式存储
master上只保存全局目录表，而数据都保存在segment上面，对于每一个表在创建时都要制定distributed策略，将满足不同策略的rows分别保存在不同的segment上面。distributed策略有两种，hash和random两种方式，默认是hash分布。对于hash分布，一般选取主键作用的列作为分布依据，如果没指定列，那么默认是主键作用的列，否则是第一列。随机分布就是将数据随机保存在不同的segment上面，在所有segment上面进行全表扫描这样的操作时，由于IO时间均摊，是个效率很高的方法。
查询处理
查询计划器(query planner)，意指在各个节点上分步骤的获取最后结果的过程，比如表扫描、join、聚集和排序操作。有的查询涉及到motion操作，即数据在segments之间迁移。对于一次查询任务，planner会把它切割成分片(slice)，对应到各个segment上去，也就是说slice之间是独立的，但一个slice可能运行在多个segment上，比如一次motion操作肯定对应两边segment上的同一个slice任务。大多数的查询任务可能最后还要将segments上产生的结果通过gather motion操作发送到master上去展现给用户，记为一次slice；也有例外，比如CREATE TABLE X AS SELECT，没有gather motion操作，不需要将数据发送到master。
并行查询执行(parallel query execution)，对于一次查询来说，首先master上建立查询分派进程(query dispatcher，QD)，负责创建和分派query planner，以及收集和展现最终结果；在segment上的工作进程叫query executor(QE)，负责完成自身任务以及和其他QE传输中间结果；对于每一个slice的执行，都是多进程作业的。query planner的每一个步骤叫做一个gangs，也就是说所有segments上的相同slice是一个gangs，gangs可能包含segments之间的数据流通信，再换句话说，一个并行查询执行，就是一个gangs接着一个gangs来完成的。
与SQL标准的比较
当前(截止3.3版本，测试版本3.2)存在的问题：
1.不支持对hash分布的key列做update操作，因为不支持hash列重分布后行数据在segments之间迁移。
2.UPDATE and DELETE statements that require data to move from one segment to another. This restricts the use of joins in update and delete operations to hash-distributed tables that have the same distribution key column(s), and the join condition must specify equality on the distribution key column(s).
--join的连接条件必须是DT列，否则需要列重分布。(确定了join的连接条件，就是确定了segments之间的连接)
3.Correlated subqueries that Greenplum’s parallel optimizer cannot internally rewrite into non-correlated joins. Most simple uses of correlated subqueries do work. Those that do not can be manually rewritten using outer joins.
--内部优化的时候，相关子查询不能重写到非相关连接里面。对于不能正常工作的相关子查询，需要手动改写到外连接里面。
4.Certain rare cases of multi-row subqueries that Greenplum’s parallel optimizer cannot internally rewrite into equijoins.
--内部优化的时候，某些罕见的多行子查询不能写成equi-join(用相等来连接的，join .. n =)的形式。
5.内部优化的时候，一些在EXISTS或NOT EXISTS子查询子句中返回的集合不能重写成连接(join)。
6.不支持子查询中UNION所有的joined tables。
7.不支持子查询中在FROM子句中返回集合函数。
8.不支持退滚游标，比如使用FETCH PRIOR,FETCH FIRST,FETCH ABSOLUTE,FETCH RELATIVE。
9.hash分布建表时，UNIQUE或PRIMARY KEY子句应该包含所有的DT列，并且建表时只能写UNIQUE子句或PRIMARY KEY子句中的一个。如果是random分布的建表，就不能写UNIQUE或PRIMARY KEY子句了。
10.CREATE UNIQUE INDEX的时候，一定要覆盖所有的DT列。random分布的不能使用该语句。
11.VOLATILE or STABLE functions cannot execute on the segments, and so are generally limited to being passed literal values as the arguments to their parameters.一些函数不能直接在segment上执行，因此限制了将字面的值传递给参数。
12.不支持触发器，因为他们依赖VOLATILE函数。
13.不强制外键约束，用户定义的外键是保存在system catalog上面的。
14.不支持序列操作函数CURRVAL和LASTVAL。
15.不支持DELETE WHERE CURRENT OF和UPDATE WHERE CURRENT OF。
不支持SQL92的特性，包括：
1.字符类型只有CHAR和VARCHAR，而不支持NCHAR或NVARCHAR类型。
2.不支持CREATE ASSERTION语句。
3.支持INTERVAL关键字，但是没依照标准。
4.不支持GET DIAGNOSTICS。
5.不支持对列GRANT INSERT/UPDATE权限。
6.不支持标准的CREATE TEMPORARY TABLE和LOCAL TEMPORARY TABLE，greenplum的临时表与Teradata的VOLATILE表同一个意思。
7.UNIQUE表述。
8.不支持MATCH PARTIAL
不支持SQL99的特性，包括：
1.不支持大数据对象CLOB、BLOB和NCLOB,但是其BYTEA和TEXT类型可以保存大数据类型。
2.不支持WITH的递归语句。
3.不支持MODULE。
4.没有CREATE PROCEDURE句法(SQL/PSM)，可以用CREATE FUNCTION代替，返回void，用法如：select my_func(args);
5.greenplum/PostgreSQL的函数定义语言(PL/PGSQL)是Oracle的PL/SQL的子集，不兼容SQL/PSM语言；greenplum还支持用python、perl和R语言实现的函数定义。
6.特意不支持BIT、BIT VARYING类型，此类型在后来的SQL标准中已被取缔。
7.greenplum支持的标识符最长63字符，而SQL标准是128字符。
8.不支持prepared transaction。
9.CHARACTER SET option on the definition of CHAR() or VARCHAR() columns
char和varchar类型对字符集有限制
10.不支持用字符或字节规定char()或varchar()列，比如VARCHAR(15 CHARACTERS) or VARCHAR(15 OCTETS) or VARCHAR(15 BYTES)
11.不支持CURRENT_SCHEMA??????(测试支持，select current_schema())
12.不支持CREATE DISTINCT TYPE语句，CREATE DOMAIN可以作为全能使用。
不支持SQL2003的特性，包括：
1.不支持XML类型，PostgreSQL支持。
2.不支持MERGE语句。
3.不支持IDENTITY列，以及GENERATED ALWAYS/GENERATED BY DEFAULT子句。
4.不支持数据类型的MULTISET修饰符。
5.不支持ROW数据类型。
6.Sequence用法不是标准的，比如nextval(‘seq’)是获取seq的下一个value。
7.不支持GENERATED ALWAYS AS列；可以用视图代替(views can be used as a work-around.)
8.在select的子句中不支持TABLESAMPLE；可以用random()代替。
9.在select和子句中不支持NULLS FIRST或NULLS LAST。
10.在表join时不支持PARTITION BY。
11.不支持对列进行grant select权限操作；可以用view实现。
12.对于CREATE TABLE X (LIKE Y)语句，不支持[INCLUDING|EXCLUDING] [DEFAULTS|CONSTRAINTS|INDEXES] 子句。
13.数组数据类型与SQL标准兼容但有例外，一般用户使用不会受影响。
不支持SQL2008的特性，包括：
1.不支持BINARY和VARBINARY类型，可以用BYTEA类型代替VARBINARY。
2.不支持FETCH FIRST、FETCH NEXT这样的句法，比如select * from sale order by id offset 20 rows fetch next 10 rows only;；可以用limit代替。
3.在VIEW和子查询中，ORDER BY最好与limit一起使用，否则会被忽略，因为可能会导致性能问题。
4.The row subquery construct is not supported.
5.TRUNCATE TABLE不支持CONTINUE IDENTITY和RESTART IDENTITY子句。
6.不支持CREATE FOREIGN DATA WRAPPER (SQL/MED)子句。
7.不支持CREATE SERVER(SQL/MED)子句。
8.不支持CREATE MAPPING(SQL/MED)子句。
参考链接
修改内核参数 (Linux)
Linux内核参数(如kernel.shmmax)及Oracle相关参数调整(如SGA_MAX_SIZE)
How to do failed Segment analysis and recovery
Recovering From Segment Failures
Postgres与OS内核相关的几个参数设置
How to build GPDB on RHEL 7 (如何在RHEL7上编译GPDB)
kenyon的Greenplum的安装
./configure --prefix=/srv/gpdb --with-libedit-preferred --with-perl --with-openssl --with-libxml --enable-thread-safety --disable-orca
对中文提示的支持到2022年2月还是没有支持，因此'--enable-nls=zh_CN'这个选项还是无效的。为其添加专用用户：
useradd -m -r -u 2345 -g daemon -s /bin/bash -c 'greenplum dbcluster user' gpdb
-------------------------------deb-------------------------------
这里是在Devuan4下安装Greenplum-6.18的相关记录
问题：缺少libreadline-dev
configure: error: readline library not found
If you have readline already installed, see config.log for details on the
failure. It is possible the compiler isn't looking in the proper directory.
Use --without-readline to disable readline support.
问题：缺少zlib1g-dev，知道新的版本支持zstd，顺带将libzstd-dev也安装了
checking for inflate in -lz... no
configure: error: zlib library not found
If you have zlib already installed, see config.log for details on the
failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zlib to disable zlib support.
问题：缺少libapr1-dev
checking for apr-1-config... no
configure: error: apr-1-config is required for gpfdist, unable to find binary
问题：缺少libevent-dev
checking for library containing apr_getopt_long... none required
checking for library containing event_add... no
configure: error: libevent is required for gpfdist
问题：缺少libyaml-dev libssl-dev
checking for library containing yaml_parser_initialize... no
configure: WARNING: libyaml is not found. disabling transformations for gpfdist.
checking for CRYPTO_new_ex_data in -lcrypto... no
configure: error: library 'crypto' is required for OpenSSL
问题：缺少libxml2-dev
checking for xmlSaveToBuffer in -lxml2... no
configure: error: library 'xml2' (version >= 2.6.23) is required for XML support
问题：缺少libcurl4-openssl-dev
checking for curl-config... no
*** The curl-config script could not be found. Make sure it is
*** in your path, and that curl is properly installed.
*** Or see http://curl.haxx.se/
configure: error: Library requirements (curl) not met.
问题：缺少libbz2-dev
checking for BZ2_bzDecompress in -lbz2... no
configure: error: library 'bz2' is required for bzip2 support
问题：缺少官方新增的优化器环境
checking for strnicmp in -lxerces-c... no
configure: error: library xerces-c is required to build with Pivotal Query Optimizer
官方提供的大数据优化器(GPORCA)，这里不需要，使用--disable-orca编译参数跳过。
问题：缺少libperl-dev
checking for libperl... no
configure: error: libperl library is required for Perl
在make的过程中还有报依赖工具(bison flex)的错
ERROR: `bison' is missing on your system. It is needed to create the
file `gram.c'. You can either get bison from a GNU mirror site
or download an official distribution of PostgreSQL, which contains
pre-packaged bison output.
一次性安装
apt-get install libreadline-dev zlib1g-dev libzstd-dev libapr1-dev libevent-dev libyaml-dev libssl-dev libxml2-dev libcurl4-openssl-dev libbz2-dev libperl-dev bison flex -y
在make install时有报python，还是安装好python吧。
/bin/sh: 5: python: not found
make[2]: *** [Makefile:80：pygresql] 错误 127
make[2]: 离开目录“/root/soft/gpdb6.18/gpMgmt/bin”
make[1]: *** [Makefile:7：install-bin-recurse] 错误 2
make[1]: 离开目录“/root/soft/gpdb6.18/gpMgmt”
make: *** [GNUmakefile:78：install] 错误 2
python3-dev，但报错中明显需要python2，真是麻烦。。。安装好python2的相关包后，可能还需要做一个名为python链接，指向python2.7。不仅master需要，所有的segment也需要安装python2。
另外一些缺省组件的代码不是自动编译的，需要单独进行编译。
-------------------------------rpm-------------------------------
在CentOS8用源码编译并安装Greenplum-6.19
问题：
checking whether g++ supports C++11 features with +std=c++0x... no
checking whether g++ supports C++11 features with -h std=c++0x... no
configure: error: *** A compiler with support for C++11 language features is required.
解决：yum install gcc-c++
问题：
configure: WARNING:
*** Without Bison you will not be able to build PostgreSQL from Git nor
*** change any of the parser definition files. You can obtain Bison from
*** a GNU mirror site. (If you are using the official distribution of
*** PostgreSQL then you do not need to worry about this, because the Bison
*** output is pre-generated.)
解决：yum install bison
问题：
configure: WARNING:
*** Without Flex you will not be able to build PostgreSQL from Git nor
*** change any of the scanner definition files. You can obtain Flex from
*** a GNU mirror site. (If you are using the official distribution of
*** PostgreSQL then you do not need to worry about this because the Flex
*** output is pre-generated.)
解决：yum install flex
问题：
checking for flags to link embedded Perl... Can't locate ExtUtils/Embed.pm in @INC (you may need to install the ExtUtils::Embed module) (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5).
BEGIN failed--compilation aborted.
no configure: error: could not determine flags for linking embedded Perl.
This probably means that ExtUtils::Embed or ExtUtils::MakeMaker is not
installed.
解决：yum install perl-ExtUtils-Embed.noarch
问题：
checking for library containing readline... no
configure: error: readline library not found
If you have readline already installed, see config.log for details on the
failure. It is possible the compiler isn't looking in the proper directory.
Use --without-readline to disable readline support.
解决：yum install readline-devel
问题：
checking for ZSTD_compressCCtx in -lzstd... no
configure: error: zstd library not found
If you have libzstd already installed, see config.log for details on the
failure. It is possible the compiler isn't looking in the proper directory.
Use --without-zstd to disable zstd support.
解决：yum install libzstd libzstd-devel
问题：
checking for library containing gss_init_sec_context... no
configure: error: could not find function 'gss_init_sec_context' required for GSSAPI
解决：yum install krb5-devel
问题：
checking for apr-1-config... no
configure: error: apr-1-config is required for gpfdist, unable to find binary
解决：yum install apr-devel
问题：
checking for library containing event_add... no
configure: error: libevent is required for gpfdist
解决：yum install libevent-devel
问题：
checking for xmlSaveToBuffer in -lxml2... no
configure: error: library 'xml2' (version >= 2.6.23) is required for XML support
解决：yum install libxml2-devel
问题：
checking for curl-config... no
*** The curl-config script could not be found. Make sure it is
*** in your path, and that curl is properly installed.
*** Or see http://curl.haxx.se/
configure: error: Library requirements (curl) not met.
解决：yum install libcurl-devel
问题：
checking for BZ2_bzDecompress in -lbz2... no
configure: error: library 'bz2' is required for bzip2 support
解决：yum install bzip2-devel
问题：
checking for strnicmp in -lxerces-c... no
configure: error: library xerces-c is required to build with Pivotal Query Optimizer
解决：yum install xerces-c-devel
make部分问题
问题：
In file included from include/reader.h:4,
from include/gpreader.h:4,
from src/gpcloud.cpp:38:
include/s3common_headers.h:8:10: 致命错误：openssl/hmac.h：No such file or directory
#include <openssl/hmac.h>
解决：yum install openssl-devel
如果安装完依赖库后，make失败，可以尝试make clean后再进行make。其安装过程与《PostgreSQL源码安装及配置》中相近。
--------------------------------------------------------------
需要生成或创立三个配置文件
--env.sh--
source /srv/gpdb/greenplum_path.sh
export PGPORT=5432
export MASTER_DATA_DIRECTORY=/home/gpdb/data/master/gpseg
--hostfile--
devf1
devf2
devf3
同样需要在操作系统的/etc/hosts中加主机相关的映射关系：
192.168.8.78 devfp
192.168.8.71 devf1
192.168.8.72 devf2
192.168.8.73 devf3
--gpinitsystem_config--
ARRAY_NAME="FreeOA GPDB"
SEG_PREFIX=gpseg
PORT_BASE=7000
# 根据需要修改下面的路径和主机名
declare -a DATA_DIRECTORY=(/home/gpdb/data /home/gpdb/data /home/gpdb/data)
MASTER_HOSTNAME=devfp
MASTER_DIRECTORY=/home/gpdb/data/master/gpseg
MASTER_PORT=5432
IP_ALLOW=0.0.0.0/0
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=3
ENCODING=UNICODE
MACHINE_LIST_FILE=./hostfile
简单说明下
ARRAY_NAME：设置阵列名称，默认Greenplum Data Platform。
SEG_PREFIX：设置segment的前缀，默认gpseg。
PORT_BASE：设置segment的起始端口，会从此端口往上增加，默认从40000开始。
DATA_DIRECTORY：设置segment primary的数据存储目录，有几个segment节点就需要设置几个数据存储目录。
MASTER_HOSTNAME：设置master的主机名。
MASTER_DIRECTORY：设置master的存储目录。
MASTER_PORT：设置master的端口，默认5432。
TRUSTED_SHELL：设置节点之间的信任方式，默认SSH。
CHECK_POINT_SEGMENTS：预写日志文件（WAL）数量，默认为8，这意味着为主机上的每个Segment或Master实例分配1088MB的WAL空间。
ENCODING=UNICODE：设置初始字符集，默认UNICODE（UTF-8）。
MACHINE_LIST_FILE：仅包含segment主机地址文件列表。
注意，其中所有需要的目录都是在创建数据存储区域时做好的。
尤其需要注意的是 MACHINE_LIST_FILE，指的是仅包含segment主机名，每行一个，主机名不要有空格。。
初始化 GPDB 集群
$ source env.sh
$ gpinitsystem -c gpinitsystem_config --locale=zh_CN.utf8 -a
注意：这里是非常简化过程，没有涉及standby与mirror这两部分。如果不出错就成功了，服务会自动起来，以后可以使用 gpstart 和 gpstop 起停服务。
初始化成功后，运行一下命令验证系统状态：
$ psql -l
$ gpstate
简单测试
--创建库
$ createdb test
--连接库
$ psql test
--使用generate_series生成表数据
test# CREATE TABLE freeoatab AS SELECT * FROM generate_series(1, 100000);
--统计每个节点的数量分布
test# SELECT gp_segment_id, count(1) FROM freeoatab GROUP BY gp_segment_id;
如果安装了新版本的优化器，可以查看其信息：
>select gp_opt_version();
gp_opt_version
---------------------------------------
Server has been compiled without ORCA
--------------------------------------------------------------
环境相关配置(操作系统)
数据库的调优所需要的知识比较广泛且有一定的深度，在这里也只是简单的提及，其后的原理就需要深入的学习了。
建议关闭SELINUX，并调整好IPTABLES中规则，下面的设置完成后，非常有必要重启该主机。分别涉及：
1.操作系统内核级参数优化(内核参数、文件、进线程数量限制、网络栈等)
2.硬盘IO优化(文件系统格式、配置参数、调度算法等)
3.时间源
4.SSH-Auto-Login
在Linux平台，推荐使用XFS文件系统，GP建议使用下面的挂载参数：rw,noatime,inode64,allocsize=16m
如在挂载XFS格式的设备/dev/sdb到目录/data，/etc/fstab中的配置如下：
/dev/sdb /data xfs rw,noatime,inode64,allocsize=16m 1 1
Linux磁盘I/O调度器对磁盘的访问支持不同的策略，默认的为CFQ，GP建议设置为deadline
要查看某驱动器的I/O调度策略，可通过如下命令查看，下面示例的为正确的配置：
# cat /sys/block/{devname}/queue/scheduler
noop anticipatory [deadline] cfq
修改磁盘I/O调度策略的方法为，修改/boot/grub/menu.lst文件的启动参数，在kernel一行的最后追加"elevator=deadline"
磁盘设备文件需要配置read-ahead(blockdev)值为65536
官方文档的推荐值为16384，但设定为65536更合理，该值设置的是预读扇区数，实际上预读的字节数是blockdev设置除以2，而GP缺省的blocksize为32KB，刚好与65536(32768B/32KB)对应。
检查某块磁盘的read-ahead设置：
# blockdev --getra devname
例如：
# blockdev --getra /dev/sda
65536
修改系统的read-ahead设置，可通过/etc/rc.d/rc.local来修改，在文件尾部追加如下代码：
# blockdev --setra 65536 /dev/mapper/vg00-LV_01
如需临时修改read-ahead设置，可通过执行下面的命令来实现：
# blockdev --setra bytes devname
例如：
# blockdev --setra 65536 /dev/sda
GP建议使用NTP(网络时间协议)来同步GPDB集群中所有主机的系统时钟。
在Segment 主机上，NTP应该配置Master 主机作为主时间源，而Standby作为备选时间源。在Master和Standby上配置NTP到首选的时间源(如果没有更好的选择可以选择Master自身作为最上端的事件源)。但要保证这组集群的时间是一致的，不能有过大的误差。软件可以使用ntpd或较新的chronyd。
--/etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072
* soft core unlimited
--/etc/sysctl.conf
kernel.shmmax = 8589934592
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.sem = 250 5120000 100 20480
kernel.sysrq = 1
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.msgmni = 2048
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.all.arp_filter = 1
net.ipv4.ip_local_port_range = 1025 65000
net.core.netdev_max_backlog = 10000
vm.overcommit_memory = 2
--------------------------------------------------------------
登录进入数据库数据中进行测试
CREATE TABLE t1 AS SELECT * FROM generate_series(1,10000000);
SELECT gp_segment_id, count(1) FROM t1 GROUP BY gp_segment_id;
create table fdfstat(
fid int4,
fsizebytes int8
)with (fillfactor=100, appendonly=true, compresstype=ZSTD, compresslevel=6) DISTRIBUTED BY ("fid");
COPY fdfstat FROM '/tmp/fdfs.149.35.220210.mist' DELIMITER ':';
GP集群信息查看
用gpstat来查看当前的状态，此命令也是排错时的必备工具：
gpstate -e #查看mirror的状态，这次的案例没有安装mirror
gpstate -f #查看standby master的状态
gpstate -s #查看整个GP群集的状态
gpstate -i #查看GP的版本
gpstate --help #帮助文档，可以查看gpstate更多用法。
性能测试
网络性能
网络测试选项包括：并行测试(-r N)、串行测试(-r n)、矩阵测试(-r M)。测试时运行一个网络测试程序从当前主机向远程主机传输5秒钟的数据流。缺省时，数据并行传输到每个远程主机，报告出传输的最小、最大、平均和中值速率，单位为MB/S。如果主体的传输速率低于预期(小于100MB/S)，可以使用-r n参数运行串行的网络测试以得到每个主机的结果。要运行矩阵测试，指定-r M参数，使得每个主机发送接收指定的所有其他主机的数据，这个测试可以验证网络层能否承受全矩阵工作负载。
gpcheckperf -f hostfile -r N -d /tmp > subnet1.out
检查磁盘IO、内存带宽
gpcheckperf -f hostfile -d /data/mirror -r ds
====================
== RESULT 2022-02-17T14:25:35.317752
====================
disk write avg time (sec): 14.24
disk write tot bytes: 50450726912
disk write tot bandwidth (MB/s): 3386.89
disk write min bandwidth (MB/s): 1060.70 [devf1]
disk write max bandwidth (MB/s): 1195.96 [devf2]
disk read avg time (sec): 13.66
disk read tot bytes: 50450726912
disk read tot bandwidth (MB/s): 3569.74
disk read min bandwidth (MB/s): 1003.62 [devf2]
disk read max bandwidth (MB/s): 1289.22 [devf3]
stream tot bandwidth (MB/s): 29537.70
stream min bandwidth (MB/s): 9406.80 [devf2]
stream max bandwidth (MB/s): 10468.40 [devf1]
检查每台Segment上的日期时间
gpssh -f hostfile -v date
关于gporca
gporca是greenplum新一代的优化器，在性能上有很大提升，在gporca即为旧式的优化(legacy)。
legacy是早期针对单节点 greenplum 而构建的，主要应用于OLTP场景，greenplum在不安装orca的情况下使用的是legacy优化器。
为什么要有两个优化器，其实都是有历史原因的，现在的greenplum使用MPP，主要应用场景变为OLAP场景；legacy对此虽然进行了修改，但是从架构设计上，使得其维护和添加新的功能越来越困难，所以有了GPORCA优化器。gporca 需要在每个节点都需要安装。
下面来列出另外一些不常见的操作问题，部分内容可能有重复。
---------------------------------------------------------------
PostgreSQL|Greenplum 启动或初始kernel SHMMAX parameter出错
原值为：33554432(32M)
可行的推荐值
1073741824(1G)
sysctl -w kernel.shmmax=1073741824
立即生效
sysctl -p
Then try to start postgresql
加到 8 GB:
$ sysctl -w kernel.shmmax=8589934592
$ sysctl -w kernel.shmall=2097152
一直有效的话，加入到 /etc/sysctl.conf 配置文件中：
kernel.shmmax = 1073741824
It occurs when either the system limit for the maximum number of semaphore sets (SEMMNI), or the system wide maximum number of semaphores (SEMMNS), would be exceeded. You need to raise the respective kernel parameter. Alternatively, reduce PostgreSQL's consumption of semaphores by reducing its max_connections parameter (currently 753).
The PostgreSQL documentation contains more information about configuring your system for PostgreSQL.",,,,,,"InternalIpcSemaphoreCreate","pg_sema.c",129,1 0x8c877c postgres errstart (elog.c:521)
sysctl -w kernel.sem="500 64000 50 150"
/etc/sysctl.conf
Add the following line:
fs.file-max=70000
在系统启动时修改semaphores空间参数
/sbin/sysctl -w kernel.sem="32000 1024000000 500 32000"
查看：
cat /proc/sys/kernel/sem
GPDB startup/initilization failed with "could not create semaphores: No space left on device"
Cause
As indicated by the error , the issue is due to improper kernel settings for semaphore and here there are not enough semaphores available at OS level.
Please NOTE: This error does not mean that you have run out of disk space.
Solution
Please refer Greenplum Database Installation Guide for recommended OS kernel parameters.
If the values are default as recommended on the Installation Guide , please verify if the values are set in double quotes, kernel will consider default values when used with quotes.
Current setting:
$ cat /etc/sysctl.conf |grep sem
kernel.sem="250 512000 100 2048"
Unable to Start PostgreSQL. kernel's SHMMAX parameter exceeded you limit
For Linux (example from the manual, to allow 16 GB):
$ sysctl -w kernel.shmmax=17179869184
$ sysctl -w kernel.shmall=4194304
---------------------------------------------------------------
所以在开始前，最好调整一下这个系统参数，不然会碰到下面的一系列问题。
env.sh
source /home/gpdb/greenplum_path.sh
export PGPORT=15432
export MASTER_DATA_DIRECTORY=/home/gpdb/data/master/gpseg1
hostfile
xd0
xd1
xd3
gpinitsystem_config
一个简单的配置文件如下：
ARRAY_NAME="FreeOA GPDB"
SEG_PREFIX=gpseg
PORT_BASE=40000
# 根据需要，修改下面的路径和主机名
declare -a DATA_DIRECTORY=(/home/gpdb/data /home/gpdb/data /home/gpdb/data)
MASTER_HOSTNAME=htcom
MASTER_DIRECTORY=/home/gpdb/data/master
MASTER_PORT=15432
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=6
ENCODING=UNICODE
MACHINE_LIST_FILE=./hostfile
调用配置文件开始初始化(gpinitsystem_config)过程中遇到的一些问题：
20170124:16:46:06:025117 gpinitsystem:htcom:postgres-[INFO]:-Commencing local /home/gpdb/bin/initdb -E UNICODE -D /home/gpdb/data/master/gpseg-1 --locale=en_US.utf8 --max_connections=250 --shared_buffers=128000kB --is_filerep_mirrored=no --backend_output=/home/gpdb/data/master/gpseg-1.initdb
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale en_US.utf8.
The default text search configuration will be set to "english".
creating directory /home/gpdb/data/master/gpseg-1 ... ok
creating subdirectories ... ok
selecting default max_connections ... initdb: error 256 from: "/home/gpdb/bin/postgres" --boot -x0 -F -c max_connections=250 -c shared_buffers=4000 -c max_fsm_pages=200000 < "/dev/null" > "/home/gpdb/data/master/gpseg-1.initdb" 2>&1
initdb: removing data directory "/home/gpdb/data/master/gpseg-1"
2017-01-24 16:46:07.122248 CST,,,p31735,th-591046880,,,,0,,,seg-1,,,,,"FATAL","XX000","could not create shared memory segment: 无效的参数 (pg_shmem.c:183)","Failed system call was shmget(key=1, size=176041872, 03600).","This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX. To reduce the request size (currently 176041872 bytes), reduce PostgreSQL's shared_buffers parameter (currently 4000) and/or its max_connections parameter (currently 253).
If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
The PostgreSQL documentation contains more information about shared memory configuration.",,,,,,"InternalIpcMemoryCreate","pg_shmem.c",183,1 0x8c877c postgres errstart (elog.c:521)
2 0x75a718 postgres <symbol not found> (pg_shmem.c:145)
3 0x75a9d2 postgres PGSharedMemoryCreate (pg_shmem.c:387)
4 0x7af330 postgres CreateSharedMemoryAndSemaphores (ipci.c:241)
5 0x8d6969 postgres BaseInit (postinit.c:542)
6 0x54e324 postgres AuxiliaryProcessMain (bootstrap.c:415)
7 0x49c4d4 postgres main (main.c:192)
8 0x7f71db153ead libc.so.6 __libc_start_main + 0xfd
9 0x49c59d postgres <symbol not found> + 0x49c59d
20170124:16:46:07:025117 gpinitsystem:htcom:postgres-[INFO]:-Start Function ERROR_EXIT
postgres@htcom:/home/gpdb$ ./bin/gpinitsystem -c gpinitsystem_config -a
...
.......................
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:------------------------------------------------
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:-Parallel process exit status
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:------------------------------------------------
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:-Total processes marked as completed = 6
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:-Total processes marked as killed = 0
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[WARN]:-Total processes marked as failed = 3 <<<<<
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:------------------------------------------------
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[FATAL]:-Errors generated from parallel processes
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:-Dumped contents of status file to the log file
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:-Building composite backout file
20170124:16:53:30:gpinitsystem:htcom:postgres-[FATAL]:-Failures detected, see log file /home/postgresql/gpAdminLogs/gpinitsystem_20170124.log for more detail Script Exiting!
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[WARN]:-Script has left Greenplum Database in an incomplete state
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[WARN]:-Run command /bin/bash /home/postgresql/gpAdminLogs/backout_gpinitsystem_postgres_20170124_165242 to remove these changes
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:-Start Function BACKOUT_COMMAND
20170124:16:53:30:031785 gpinitsystem:htcom:postgres-[INFO]:-End Function BACKOUT_COMMAND
20170124:17:31:14:032318 gpstart:htcom:postgres-[DEBUG]:-[worker3] finished cmd: remote segment starts on host 'xd3' cmdStr='ssh -o 'StrictHostKeyChecking no' xd3 ". /home/gpdb/greenplum_path.sh; $GPHOME/sbin/gpsegstart.py -C en_US.utf8:en_US.utf8:en_US.utf8 -M mirrorless -V 'postgres (Greenplum Database) 5.0.0 build dev' -n 9 --era 39a9ae0cf4490730_170124173112 -t 600 -v -D '8|6|p|p|s|u|xd3|xd3|40000|None|/home/gpdb/data/gpseg6||' -D '9|7|p|p|s|u|xd3|xd3|40001|None|/home/gpdb/data/gpseg7||' -D '10|8|p|p|s|u|xd3|xd3|40002|None|/home/gpdb/data/gpseg8||' -l '/home/postgresql/gpAdminLogs'"' had result: cmd had rc=1 completed=True halted=False
stdout=''
stderr='Error: unable to import module: No module named psutil
由于它的初始脚本是用python开发的，而这个工具脚本运行过程中需要一些第三方的模块，因此就必须先安装这些模块：
pip install psi lockfile paramiko setuptools epydoc psutil
在初始化greenplum时，报了一个错误
20161012:14:58:57:005531 gpcreateseg.sh:master:gpadmin-[INFO]:-Start Function ED_PG_CONF
20161012:14:58:57:005531 gpcreateseg.sh:master:gpadmin-[WARN]:-Failed to insert port=40000 in /data/gpadmin/gpdata/primary/gpseg2/postgresql.conf on sdw3
20161012:14:58:57:005531 gpcreateseg.sh:master:gpadmin-[INFO]:-End Function ED_PG_CONF
20161012:14:58:57:005531 gpcreateseg.sh:master:gpadmin-[FATAL][2]:-Failed Update port number to 40000
网上搜了一下，需要安装ed
yum install -y ed
官方文档没有提及这个依赖包，故此需要注意一下，而且每个节点包含segment都要把ed安装一遍
将程序文件从主节点上复制一份到所有的子节点：
rsync -av --exclude=master /home/gpdb/ xd3:/home/gpdb/
---------------------------------------------------------------
在家中的crux linux上的实验过程
crux上的openssh没有ssh-copy-id工具，需要手动操作一下：
ssh postgres@crux1 'mkdir -p ~/.ssh'
cat ~/.ssh/id_rsa.pub | ssh postgres@crux1 'cat >> ~/.ssh/authorized_keys'
安装过程：
postgres@pserv:/home/gpdb$ bin/gpinitsystem -c gpinit_system_config --max_connections=64 --shared_buffers=256MB
bin/lib/gp_bash_functions.sh: line 77: return: Problem in gp_bash_functions, command 'netstat' not found in COMMAND path. You will need to edit the script named gp_bash_functions.sh to properly locate the needed commands for your platform.: numeric argument required
20170225:20:02:16:000351 gpinitsystem:pserv:postgres-[INFO]:-Checking configuration parameters, please wait...
20170225:20:02:16:000351 gpinitsystem:pserv:postgres-[INFO]:-Reading Greenplum configuration file gpinit_system_config
20170225:20:02:16:000351 gpinitsystem:pserv:postgres-[INFO]:-Locale has not been set in gpinit_system_config, will set to default value
20170225:20:02:16:000351 gpinitsystem:pserv:postgres-[FATAL]:-Unable to locate locale value en_US.utf8 on this host
20170225:20:02:17:000351 gpinitsystem:pserv:postgres-[INFO]:-Select another locale value via -n option to this utility
20170225:20:02:17:000351 gpinitsystem:pserv:postgres-[INFO]:-Available locale values have been dumped to the log file
20170225:20:02:17:gpinitsystem:pserv:postgres-[FATAL]:-Unable to continue Script Exiting!
我机器上的locale有：
postgres@pserv:/home/gpdb$ locale -a
C
POSIX
zh_CN.utf8
手动生成一下en_US的locale
localedef -i en_US -f UTF-8 en_US.UTF-8
postgres@pserv:/home/gpdb$ bin/gpinitsystem -c gpinit_system_config --max_connections=64 --shared_buffers=256MB
bin/lib/gp_bash_functions.sh: line 77: return: Problem in gp_bash_functions, command 'netstat' not found in COMMAND path. You will need to edit the script named gp_bash_functions.sh to properly locate the needed commands for your platform.: numeric argument required
20170225:20:31:31:002137 gpinitsystem:pserv:postgres-[INFO]:-Checking configuration parameters, please wait...
20170225:20:31:31:002137 gpinitsystem:pserv:postgres-[INFO]:-Reading Greenplum configuration file gpinit_system_config
20170225:20:31:31:002137 gpinitsystem:pserv:postgres-[INFO]:-Locale has not been set in gpinit_system_config, will set to default value
20170225:20:31:31:002137 gpinitsystem:pserv:postgres-[INFO]:-Locale set to en_US.utf8
20170225:20:31:32:002137 gpinitsystem:pserv:postgres-[INFO]:-No DATABASE_NAME set, will exit following template1 updates
20170225:20:31:32:002137 gpinitsystem:pserv:postgres-[WARN]:-Master open file limit is 1024 should be >= 65535
20170225:20:31:32:002137 gpinitsystem:pserv:postgres-[INFO]:-Checking configuration parameters, Completed
20170225:20:31:32:002137 gpinitsystem:pserv:postgres-[INFO]:-Commencing multi-home checks, please wait...
...
20170225:20:31:34:002137 gpinitsystem:pserv:postgres-[INFO]:-Configuring build for standard array
20170225:20:31:34:002137 gpinitsystem:pserv:postgres-[INFO]:-Commencing multi-home checks, Completed
20170225:20:31:34:002137 gpinitsystem:pserv:postgres-[INFO]:-Building primary segment instance array, please wait...
.........
20170225:20:31:42:002137 gpinitsystem:pserv:postgres-[INFO]:-Building group mirror array type , please wait...
.........
20170225:20:31:50:002137 gpinitsystem:pserv:postgres-[INFO]:-Checking Master host
20170225:20:31:50:002137 gpinitsystem:pserv:postgres-[INFO]:-Checking new segment hosts, please wait...
20170225:20:31:51:002137 gpinitsystem:pserv:postgres-[WARN]:-Host pserv open files limit is 1024 should be >= 65535
20170225:20:31:52:gpinitsystem:pserv:postgres-[FATAL]:-No /home/gpdb/lib on segment instance crux1 Script Exiting!
crux下没有netstat指令了，取而代之的是ss，在shell的初始文件中做一个别名吧：
alias netstat='/sbin/ss'
将master机器上的程序文件复制到各个节点上
rsync -av --exclude=master /home/gpdb/ crux[1-3]:/home/gpdb/
Warning: Permanently added 'crux1,192.168.8.11' (ECDSA) to the list of known hosts.
Permission denied (publickey,password,keyboard-interactive).
[FATAL]:-Unexpected EOF on RemotePysync output stream
Permission denied (publickey,password,keyboard-interactive).
Permission denied (publickey,password,keyboard-interactive).
[FATAL]:-Unexpected EOF on RemotePysync output stream
[FATAL]:-Unexpected EOF on RemotePysync output stream
20170225:20:37:57:010898 gpcreateseg.sh:pserv:postgres-[FATAL]:- Command export GPHOME=/home/gpdb; . /home/gpdb/greenplum_path.sh; /home/gpdb/bin/lib/pysync.py -x pg_log -x postgresql.conf -
x postmaster.pid /home/gpdb/primary/gpseg7 \[crux1\]:/home/gpdb/mirror/gpseg7 on crux3 failed with error status 3
20170225:20:37:57:011228 gpcreateseg.sh:pserv:postgres-[FATAL]:- Command export GPHOME=/home/gpdb; . /home/gpdb/greenplum_path.sh; /home/gpdb/bin/lib/pysync.py -x pg_log -x postgresql.conf -
x postmaster.pid /home/gpdb/primary/gpseg8 \[crux1\]:/home/gpdb/mirror/gpseg8 on crux3 failed with error status 3
20170225:20:37:57:010590 gpcreateseg.sh:pserv:postgres-[FATAL]:- Command export GPHOME=/home/gpdb; . /home/gpdb/greenplum_path.sh; /home/gpdb/bin/lib/pysync.py -x pg_log -x postgresql.conf -
x postmaster.pid /home/gpdb/primary/gpseg6 \[crux1\]:/home/gpdb/mirror/gpseg6 on crux3 failed with error status 3
20170225:20:37:57:010898 gpcreateseg.sh:pserv:postgres-[INFO]:-End Function RUN_COMMAND_REMOTE
20170225:20:37:57:010590 gpcreateseg.sh:pserv:postgres-[INFO]:-End Function RUN_COMMAND_REMOTE
20170225:20:37:57:011228 gpcreateseg.sh:pserv:postgres-[INFO]:-End Function RUN_COMMAND_REMOTE
20170225:20:37:57:010898 gpcreateseg.sh:pserv:postgres-[FATAL][7]:-Failed remote copy of segment data directory from crux3 to crux1
20170225:20:37:57:011228 gpcreateseg.sh:pserv:postgres-[FATAL][8]:-Failed remote copy of segment data directory from crux3 to crux1
20170225:20:37:57:010590 gpcreateseg.sh:pserv:postgres-[FATAL][6]:-Failed remote copy of segment data directory from crux3 to crux1
20170225:20:37:58:008423 gpinitsystem:pserv:postgres-[INFO]:-End Function PARALLEL_WAIT
20170225:20:37:58:008423 gpinitsystem:pserv:postgres-[INFO]:-End Function PARALLEL_COUNT
20170225:20:37:58:008423 gpinitsystem:pserv:postgres-[INFO]:-Start Function PARALLEL_SUMMARY_STATUS_REPORT
20170225:20:37:58:008423 gpinitsystem:pserv:postgres-[INFO]:------------------------------------------------
20170225:20:37:58:008423 gpinitsystem:pserv:postgres-[INFO]:-Parallel process exit status
20170225:20:37:58:008423 gpinitsystem:pserv:postgres-[INFO]:------------------------------------------------
20170225:20:37:59:008423 gpinitsystem:pserv:postgres-[INFO]:-Total processes marked as completed = 0
20170225:20:37:59:008423 gpinitsystem:pserv:postgres-[INFO]:-Total processes marked as killed = 0
20170225:20:37:59:008423 gpinitsystem:pserv:postgres-[WARN]:-Total processes marked as failed = 9 <<<<<
20170225:20:37:59:008423 gpinitsystem:pserv:postgres-[INFO]:------------------------------------------------
20170225:20:37:59:008423 gpinitsystem:pserv:postgres-[INFO]:-End Function PARALLEL_SUMMARY_STATUS_REPORT
FAILED:crux2~50002~/home/gpdb/mirror/gpseg2~13~2~51002
FAILED:crux2~50000~/home/gpdb/mirror/gpseg0~11~0~51000
FAILED:crux2~50001~/home/gpdb/mirror/gpseg1~12~1~51001
FAILED:crux3~50001~/home/gpdb/mirror/gpseg4~15~4~51001
FAILED:crux3~50000~/home/gpdb/mirror/gpseg3~14~3~51000
FAILED:crux3~50002~/home/gpdb/mirror/gpseg5~16~5~51002
FAILED:crux1~50001~/home/gpdb/mirror/gpseg7~18~7~51001
FAILED:crux1~50000~/home/gpdb/mirror/gpseg6~17~6~51000
FAILED:crux1~50002~/home/gpdb/mirror/gpseg8~19~8~51002
20170225:20:37:59:008423 gpinitsystem:pserv:postgres-[INFO]:-End Function CREATE_QES_MIRROR
看来需要将各个segment之间都要做成ssh-autologin
在集群完成创建后，开始破坏性的测试：在insert|update时，将其中的一台segment主机reboot。
test=# insert into wnb select * from wnqw;
NOTICE: Releasing segworker groups to finish aborting the transaction.
ERROR: FTS detected connection lost during dispatch to seg0 192.168.8.11:40000 pid=3610:
ERROR: Command could not be dispatch to segment seg0 192.168.8.11:40000 pid=3610: another command is already in progress
时间：154611.104 ms
这期间将一台seg重启了，下面也是如此：
test=# update wnqw set status=2 where id between 109491419 and 110491418;
WARNING: mirror failure, could not complete mirrored request identifier 'base/16384/16420.1' ack state 'waiting for ack', failover requested (seg5 192.168.8.12:40002 pid=7127)
提示: run gprecoverseg to re-establish mirror connectivity
WARNING: mirror failure, could not complete operation on mirror, failover requested (seg5 192.168.8.12:40002 pid=7127)
描述: identifier 'base/16384/16420.1' operation 'not specified' relation type 'append only' message count '-1'
提示: run gprecoverseg to re-establish mirror connectivity
NOTICE: Releasing segworker groups to finish aborting the transaction.
ERROR: FTS detected connection lost during dispatch to seg6 192.168.8.13:40000 pid=6149:
ERROR: Command could not be dispatch to segment seg6 192.168.8.13:40000 pid=6149: another command is already in progress
时间：75945.877 ms
postgres@pserv:/home/gpdb$ bin/gpstate -m
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:-Starting gpstate with args: -m
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.0.0 build dev'
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.0.0 build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (CRUX-x86_64-multilib) 5.2.0 compiled on Feb 2 2017 18:39:11'
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:-Obtaining Segment details from master...
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:--Current GPDB mirror list and status
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:--Type = Group
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- Mirror Datadir Port Status Data Status
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg0 50000 Passive Synchronized
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg1 50001 Passive Synchronized
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg2 50002 Passive Synchronized
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg3 50000 Passive Synchronized
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg4 50001 Passive Synchronized
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg5 50002 Passive Synchronized
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg6 50000 Passive Resynchronizing
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg7 50001 Passive Resynchronizing
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg8 50002 Passive Resynchronizing
20170226:17:48:47:008824 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
修改成功后，再选一台seg主机再reboot。
test=# update wnqw set update_time=now() where id>109491418;
WARNING: mirror failure, could not queue message to be mirrored, failover requested (seg0 192.168.8.11:40000 pid=4198)
描述: identifier 'base/16384/16420.1' operation 'write' relation type 'append only' message count '-1'
提示: run gprecoverseg to re-establish mirror connectivity
NOTICE: Releasing segworker groups to finish aborting the transaction.
ERROR: FTS detected connection lost during dispatch to seg3 192.168.8.12:40000 pid=8560:
ERROR: Command could not be dispatch to segment seg3 192.168.8.12:40000 pid=8560: another command is already in progress
时间：800617.362 ms
test=# update wnb set update_time=now() where id>110091418;
NOTICE: Releasing segworker groups to finish aborting the transaction.
ERROR: FTS detected connection lost during dispatch to seg0 192.168.8.11:40000 pid=5997:
ERROR: Command could not be dispatch to segment seg0 192.168.8.11:40000 pid=5997: another command is already in progress
时间：114056.240 ms
不做恢复再更新一次
test=# update wnb set update_time=now() where id>110091418;
UPDATE 3400024
时间：177193.271 ms
test=# select min(update_time),max(update_time) from wnb where id>110091418;
min | max
----------------------------+----------------------------
2017-02-26 18:41:34.061123 | 2017-02-26 18:41:34.061123
(1 行记录)
时间：27466.814 ms
postgres@pserv:/home/gpdb$ bin/gpstate -m
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:-Starting gpstate with args: -m
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.0.0 build dev'
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.0.0 build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (CRUX-x86_64-multilib) 5.2.0 compiled on Feb 2 2017 18:39:11'
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:-Obtaining Segment details from master...
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:--Current GPDB mirror list and status
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:--Type = Group
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- Mirror Datadir Port Status Data Status
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg0 50000 Acting as Primary Resynchronizing
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg1 50001 Acting as Primary Resynchronizing
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg2 50002 Acting as Primary Resynchronizing
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg3 50000 Passive Synchronized
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg4 50001 Passive Synchronized
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg5 50002 Passive Synchronized
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg6 50000 Passive Resynchronizing
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg7 50001 Passive Resynchronizing
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg8 50002 Passive Synchronized
20170226:18:52:23:011124 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
20170226:18:52:23:011124 gpstate:pserv:postgres-[WARNING]:-3 segment(s) configured as mirror(s) are acting as primaries
postgres@pserv:/home/gpdb$ bin/gpstate -m
20170226:19:48:12:013555 gpstate:pserv:postgres-[INFO]:-Starting gpstate with args: -m
20170226:19:48:12:013555 gpstate:pserv:postgres-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.0.0 build dev'
20170226:19:48:12:013555 gpstate:pserv:postgres-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.0.0 build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (CRUX-x86_64-multilib) 5.2.0 compiled on Feb 2 2017 18:39:11'
20170226:19:48:12:013555 gpstate:pserv:postgres-[INFO]:-Obtaining Segment details from master...
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:--Current GPDB mirror list and status
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:--Type = Group
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- Mirror Datadir Port Status Data Status
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg0 50000 Passive Synchronized
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg1 50001 Passive Synchronized
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux2 /home/gpdb/mirror/gpseg2 50002 Acting as Primary Change Tracking
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg3 50000 Passive Resynchronizing
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg4 50001 Passive Resynchronizing
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux3 /home/gpdb/mirror/gpseg5 50002 Passive Resynchronizing
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg6 50000 Acting as Primary Resynchronizing
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg7 50001 Acting as Primary Resynchronizing
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:- crux1 /home/gpdb/mirror/gpseg8 50002 Acting as Primary Resynchronizing
20170226:19:48:13:013555 gpstate:pserv:postgres-[INFO]:--------------------------------------------------------------
20170226:19:48:13:013555 gpstate:pserv:postgres-[WARNING]:-4 segment(s) configured as mirror(s) are acting as primaries
20170226:19:48:13:013555 gpstate:pserv:postgres-[WARNING]:-1 mirror segment(s) acting as primaries are in change tracking
postgres@pserv:/home/gpdb$ bin/gpstate
20170226:19:48:41:013617 gpstate:pserv:postgres-[INFO]:-Starting gpstate with args:
20170226:19:48:41:013617 gpstate:pserv:postgres-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.0.0 build dev'
20170226:19:48:41:013617 gpstate:pserv:postgres-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.0.0 build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (CRUX-x86_64-multilib) 5.2.0 compiled on Feb 2 2017 18:39:11'
20170226:19:48:41:013617 gpstate:pserv:postgres-[INFO]:-Obtaining Segment details from master...
20170226:19:48:41:013617 gpstate:pserv:postgres-[INFO]:-Gathering data from segments...
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:-Greenplum instance status summary
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:-----------------------------------------------------
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Master instance = Active
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Master standby = No master standby configured
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total segment instance count from metadata = 18
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:-----------------------------------------------------
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Primary Segment Status
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:-----------------------------------------------------
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total primary segments = 9
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total primary segment valid (at master) = 8
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total primary segment failures (at master) = 1 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number of postmaster.pid files missing = 9 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number of postmaster.pid files found = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number of postmaster.pid PIDs missing = 9 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number of postmaster.pid PIDs found = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number of /tmp lock files missing = 9 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number of /tmp lock files found = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number postmaster processes missing = 9 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number postmaster processes found = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:-----------------------------------------------------
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Mirror Segment Status
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:-----------------------------------------------------
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total mirror segments = 9
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total mirror segment valid (at master) = 9
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total mirror segment failures (at master) = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number of postmaster.pid files missing = 9 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number of postmaster.pid files found = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number of postmaster.pid PIDs missing = 9 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number of postmaster.pid PIDs found = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number of /tmp lock files missing = 9 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number of /tmp lock files found = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number postmaster processes missing = 9 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number postmaster processes found = 0
20170226:19:48:42:013617 gpstate:pserv:postgres-[WARNING]:-Total number mirror segments acting as primary segments = 4 <<<<<<<<
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:- Total number mirror segments acting as mirror segments = 5
20170226:19:48:42:013617 gpstate:pserv:postgres-[INFO]:-----------------------------------------------------
postgres@htcom:/home/gpdb$ bin/gprecoverseg
postgres@htcom:/home/gpdb$ bin/gpstate -s
postgres@htcom:/home/gpdb$ bin/gpstate -m
postgres@htcom:/home/gpdb$ bin/gprecoverseg -r
20170227:16:43:53:008719 gprecoverseg:htcom:postgres-[INFO]:-Starting gprecoverseg with args: -r
20170227:16:43:53:008719 gprecoverseg:htcom:postgres-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.0.0 build dev'
20170227:16:43:53:008719 gprecoverseg:htcom:postgres-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.0.0 build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (Debian 4.7.2-5) 4.7.2 compiled on Jan 24 2017 16:17:20'
20170227:16:43:54:008719 gprecoverseg:htcom:postgres-[INFO]:-Checking if segments are ready to connect
20170227:16:43:54:008719 gprecoverseg:htcom:postgres-[INFO]:-Obtaining Segment details from master...
20170227:16:43:54:008719 gprecoverseg:htcom:postgres-[INFO]:-Obtaining Segment details from master...
20170227:16:43:54:008719 gprecoverseg:htcom:postgres-[CRITICAL]:-gprecoverseg failed. (Reason='Down segments still exist. All segments must be up to rebalance.') exiting...
不是每次都能恢复成功，有时要执行gpstop，后又gpstart。主节点大都能起来，数据的安全性还是有保障的。
-------------------------------
20170323:15:46:10:001297 gpinitsystem:htcom:postgres-[WARN]:-Host htcom open files limit is 1024 should be >= 65535
这个文件最大打开数不只是调整系统的，还要调对应用户的。
The database cluster will be initialized with locale en_US.utf8.
The default text search configuration will be set to "english".
creating directory /home/gpdb/master/gpseg-1 ... ok
creating subdirectories ... ok
selecting default max_connections ... initdb: error 256 from: "/home/gpdb/bin/postgres" --boot -x0 -F -c max_connections=250 -c shared_buffers=4000 -c max_fsm_pages=200000 < "/dev/null" > "/home/gpdb/master/gpseg-1.initdb" 2>&1
initdb: removing data directory "/home/gpdb/master/gpseg-1"
2017-03-23 15:54:03.548500 CST,,,p8061,th250627872,,,,0,,,seg-1,,,,,"FATAL","XX000","could not create shared memory segment: 无效的参数 (pg_shmem.c:183)","Failed system call was shmget(key=1, size=176041872, 03600).","This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX. To reduce the request size (currently 176041872 bytes), reduce PostgreSQL's shared_buffers parameter (currently 4000) and/or its max_connections parameter (currently 253).
If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
The PostgreSQL documentation contains more information about shared memory configuration.",,,,,,"InternalIpcMemoryCreate","pg_shmem.c",183,1 0x8c877c postgres errstart (elog.c:521)
2 0x75a718 postgres <symbol not found> (pg_shmem.c:145)
3 0x75a9d2 postgres PGSharedMemoryCreate (pg_shmem.c:387)
4 0x7af330 postgres CreateSharedMemoryAndSemaphores (ipci.c:241)
5 0x8d6969 postgres BaseInit (postinit.c:542)
6 0x54e324 postgres AuxiliaryProcessMain (bootstrap.c:415)
7 0x49c4d4 postgres main (main.c:192)
8 0x7f4c0d402ead libc.so.6 __libc_start_main + 0xfd
9 0x49c59d postgres <symbol not found> + 0x49c59d
20170323:15:54:03:003438 gpinitsystem:htcom:postgres-[INFO]:-Start Function ERROR_EXIT
20170323:15:54:03:gpinitsystem:htcom:postgres-[FATAL]:- Command /home/gpdb/bin/initdb -E UNICODE -D /home/gpdb/master/gpseg-1 --locale=en_US.utf8 --max_connections=250 --shared_buffers=128000kB --is_filerep_mirrored=no --backend_output=/home/gpdb/master/gpseg-1.initdb failed with error status 1, see log file /home/postgres/gpAdminLogs/gpinitsystem_20170323.log for more detail Script Exiting!
其中的一个节点不能初始成功：
...
creating template1 database in /mnt/pc/gpdb/gpseg1/base/1 ... 2017-03-23 16:01:18.635500 CST,,,p29513,th-1561041120,,,,0,,,seg-1,,,,,"FATAL","XX000","could not create semaphores:
设备上没有空间 (pg_sema.c:129)","Failed system call was semget(111, 17, 03600).","This error does *not* mean that you have run out of disk space.
It occurs when either the system limit for the maximum number of semaphore sets (SEMMNI), or the system wide maximum number of semaphores (SEMMNS), would be exceeded. You need to raise the respective kernel parameter. Alternatively, reduce PostgreSQL's consumption of semaphores by reducing its max_connections parameter (currently 753).
The PostgreSQL documentation contains more information about configuring your system for PostgreSQL.",,,,,,"InternalIpcSemaphoreCreate","pg_sema.c",129,1 0x8c877c postgres errst
art (elog.c:521)
2 0x75a1b9 postgres PGSemaphoreCreate (pg_sema.c:114)
3 0x7b9df1 postgres InitProcGlobal (proc.c:216)
4 0x7af5bc postgres CreateSharedMemoryAndSemaphores (ipci.c:340)
5 0x8d6969 postgres BaseInit (postinit.c:542)
6 0x54e324 postgres AuxiliaryProcessMain (bootstrap.c:415)
7 0x49c4d4 postgres main (main.c:192)
8 0x7f69a1444ead libc.so.6 __libc_start_main + 0xfd
9 0x49c59d postgres <symbol not found> + 0x49c59d
child process exited with exit code 1
initdb: removing data directory "/mnt/pc/gpdb/gpseg1"
20170323:16:01:18:028704 gpcreateseg.sh:htcom:postgres-[INFO]:-Start Function BACKOUT_COMMAND
20170323:16:01:18:028704 gpcreateseg.sh:htcom:postgres-[INFO]:-End Function BACKOUT_COMMAND
20170323:16:01:18:028704 gpcreateseg.sh:htcom:postgres-[INFO]:-Start Function BACKOUT_COMMAND
20170323:16:01:18:028704 gpcreateseg.sh:htcom:postgres-[INFO]:-End Function BACKOUT_COMMAND
20170323:16:01:18:028704 gpcreateseg.sh:htcom:postgres-[FATAL][1]:-Failed to start segment instance database htcom /mnt/pc/gpdb/gpseg1
ok
loading file-system persistent tables for template1 (mirrored = false) ...
ok
loading file-system persistent tables for template1 (mirrored = false) ...
ok
...
还是老问题，要调整内核参数(crux上的这个参数设置的比较合理，基本不用做调整，下面这个就是crux上的参数值)：
/sbin/sysctl -w kernel.sem="32000 1024000000 500 32000"
创建支持压缩表的方法
CREATE TABLE "wnm2qz" (
...
)
WITH (appendonly=true, compresstype=QUICKLZ, compresslevel=1)
DISTRIBUTED BY ("id");
WITH (appendonly=true, compresstype=QUICKLZ, compresslevel=7)
[Err] ERROR: compresslevel=7 is out of range for quicklz (should be 1)
才发现系统中根本没有其链接库
ERROR: 22000: quicklz compression not supported (quicklz_compression.c:26) (seg2 192.168.0.9:40002 pid=21991)
位置: cdbdisp_finishCommand, cdbdisp.c:239
CREATE TABLE "wnm2z" (
...
)
WITH (appendonly=true, compresstype=ZLIB, compresslevel=6)
DISTRIBUTED BY ("id");
-------------------------------
个人的使用感受
本人是一名运维人员，只是想测试一下'大量'数据的情况下查询的效率。个人认为它很优秀的地方有：
不错的在线压缩功能
按特定条件的分布存放，分区管理
充分利用它所在的每台机器的资源
数据的安全性有保障的且恢复不难
一些不错的功能点：
单一入口
支持列存储
个人更喜欢在单机上玩伪分布式的方式，将机器的存储方式改为多组阵列1(raid1)。
mkdir -p /home/gpdb/primary /home/gpdb/mirror && chown -R postgres.daemon /home/gpdb
其配置文件也更简单，下面就是我的这种方式的配置文件：
ARRAY_NAME="FreeOA GPDB"
SEG_PREFIX=gpseg
PORT_BASE=40000
declare -a DATA_DIRECTORY=(/mnt/pb/gpdb /mnt/pc/gpdb /mnt/pd/gpdb)
MASTER_PORT=15432
MASTER_HOSTNAME=htcom
MASTER_DIRECTORY=/home/gpdb/master
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=8
ENCODING=UNICODE
MACHINE_LIST_FILE=./hostfile
hostfile也只需写本机名称即可
hto@htcom:/home/gpdb$ more hostfile
htcom
greenplum的备份，提供了gp_dump做并行备份，master和segment节点同时执行备份操作，另外gp_crondump会定期执行备份操作；支持PostgreSQL的pg_dump和pg_dumpall命令，但是其将所有的数据写成一个文件保存在master上，使用受限。greenplum数据库的启动gpstart，关闭gpstop，重启gpstop -r，强制关闭gpstop -f ，修改参数后不重启DB加载参数gpstop -u ，仅启动mastergpstart -m；
系统架构上一些理解
冗余和镜像：
基于PostgreSQL 8.2.13，内部修改以支持并行结构，segments上保存数据，节点间内部通信默认采用UDP协议，用额外的packet负责控制信息，代替TCP协议且取得不错的效果。
segment镜像和segment一定保存在不同的host上，master如果连不上segment，就把它标记为invalid状态，如果下次连上了，就重置为valid状态。如果系统没配置镜像，那么master在检测到invalid的segment时就会关闭数据库，来保证数据不出错；如果系统配置了镜像，那么系统在read-only模式和continue模式下的处理方式是不同的，前者不允许DDL和DML操作，可在线恢复，后者的操作必须限制在非invalid的segment上的数据，而且invalid的segment恢复时必须重启数据库系统。
master镜像就是master的在线standby，因为master上面只保存目录表和系统日志，因此在standby上面有一个事务重复记录进程(gpsyncagent)连接master以同步，一旦master连不上即坏掉，该进程停止工作，并让standby从master上一个成功事务操作之后开始接管系统(PS:master出故障前，standby上面只有gpsyncagent一个进程运行)。互连冗余，意指每个segment至少双网卡，一用网络连接，二用内部数据传输，提高通信可靠性。(详细参考白皮书的第五章系统管理的高可用性章节)
数据并行加载：
通过外部表(external table)，使用并行文件服务器(gpfdist)，可实现并行的数据加载。策略是M:N，即每个segment都会去连接每个gpfdist服务器。可通过postgresql.conf配置文件的gp_external_max_segs参数调整连接一个gpfdist服务器的segments个数，默认是64。
换句话说，external tables就是数据库将text文件按照列分割展示为数据库表的形式，可以执行select、join、sort这样的SQL操作了，相当于ETL的“抽取”部分。当然这个表是read-only的，不能执行DML操作，也不能建索引，并且数据展示依赖于gpfdist服务进程的存在。最后可以用CREATE TABLE X AS SELECT和INSERT INTO SELECT这样的语句实现ETL的“加载”部分，在加载的性能上，要考虑以下3点：先导入后建索引；导入后执行analyze更新query planner记录的信息；导入时遇到错误的话，最后要执行vacuum，用来释放失败的导入操作占用的空间。
管理和监视
简单说，管理包括从数据库安装到数据并行加载的很多命令，都保存在$GPHOME/bin目录下。要想对greenplum进行性能监视，就要在每个host上安装一个agent，一般每15秒采集一次数据，发送到master上。
分布式存储
master上只保存全局目录表，而数据都保存在segment上面，对于每一个表在创建时都要制定distributed策略，将满足不同策略的rows分别保存在不同的segment上面。distributed策略有两种，hash和random两种方式，默认是hash分布。对于hash分布，一般选取主键作用的列作为分布依据，如果没指定列，那么默认是主键作用的列，否则是第一列。随机分布就是将数据随机保存在不同的segment上面，在所有segment上面进行全表扫描这样的操作时，由于IO时间均摊，是个效率很高的方法。
查询处理
查询计划器(query planner)，意指在各个节点上分步骤的获取最后结果的过程，比如表扫描、join、聚集和排序操作。有的查询涉及到motion操作，即数据在segments之间迁移。对于一次查询任务，planner会把它切割成分片(slice)，对应到各个segment上去，也就是说slice之间是独立的，但一个slice可能运行在多个segment上，比如一次motion操作肯定对应两边segment上的同一个slice任务。大多数的查询任务可能最后还要将segments上产生的结果通过gather motion操作发送到master上去展现给用户，记为一次slice；也有例外，比如CREATE TABLE X AS SELECT，没有gather motion操作，不需要将数据发送到master。
并行查询执行(parallel query execution)，对于一次查询来说，首先master上建立查询分派进程(query dispatcher，QD)，负责创建和分派query planner，以及收集和展现最终结果；在segment上的工作进程叫query executor(QE)，负责完成自身任务以及和其他QE传输中间结果；对于每一个slice的执行，都是多进程作业的。query planner的每一个步骤叫做一个gangs，也就是说所有segments上的相同slice是一个gangs，gangs可能包含segments之间的数据流通信，再换句话说，一个并行查询执行，就是一个gangs接着一个gangs来完成的。
与SQL标准的比较
当前(截止3.3版本，测试版本3.2)存在的问题：
1.不支持对hash分布的key列做update操作，因为不支持hash列重分布后行数据在segments之间迁移。
2.UPDATE and DELETE statements that require data to move from one segment to another. This restricts the use of joins in update and delete operations to hash-distributed tables that have the same distribution key column(s), and the join condition must specify equality on the distribution key column(s).
--join的连接条件必须是DT列，否则需要列重分布。(确定了join的连接条件，就是确定了segments之间的连接)
3.Correlated subqueries that Greenplum’s parallel optimizer cannot internally rewrite into non-correlated joins. Most simple uses of correlated subqueries do work. Those that do not can be manually rewritten using outer joins.
--内部优化的时候，相关子查询不能重写到非相关连接里面。对于不能正常工作的相关子查询，需要手动改写到外连接里面。
4.Certain rare cases of multi-row subqueries that Greenplum’s parallel optimizer cannot internally rewrite into equijoins.
--内部优化的时候，某些罕见的多行子查询不能写成equi-join(用相等来连接的，join .. n =)的形式。
5.内部优化的时候，一些在EXISTS或NOT EXISTS子查询子句中返回的集合不能重写成连接(join)。
6.不支持子查询中UNION所有的joined tables。
7.不支持子查询中在FROM子句中返回集合函数。
8.不支持退滚游标，比如使用FETCH PRIOR,FETCH FIRST,FETCH ABSOLUTE,FETCH RELATIVE。
9.hash分布建表时，UNIQUE或PRIMARY KEY子句应该包含所有的DT列，并且建表时只能写UNIQUE子句或PRIMARY KEY子句中的一个。如果是random分布的建表，就不能写UNIQUE或PRIMARY KEY子句了。
10.CREATE UNIQUE INDEX的时候，一定要覆盖所有的DT列。random分布的不能使用该语句。
11.VOLATILE or STABLE functions cannot execute on the segments, and so are generally limited to being passed literal values as the arguments to their parameters.一些函数不能直接在segment上执行，因此限制了将字面的值传递给参数。
12.不支持触发器，因为他们依赖VOLATILE函数。
13.不强制外键约束，用户定义的外键是保存在system catalog上面的。
14.不支持序列操作函数CURRVAL和LASTVAL。
15.不支持DELETE WHERE CURRENT OF和UPDATE WHERE CURRENT OF。
不支持SQL92的特性，包括：
1.字符类型只有CHAR和VARCHAR，而不支持NCHAR或NVARCHAR类型。
2.不支持CREATE ASSERTION语句。
3.支持INTERVAL关键字，但是没依照标准。
4.不支持GET DIAGNOSTICS。
5.不支持对列GRANT INSERT/UPDATE权限。
6.不支持标准的CREATE TEMPORARY TABLE和LOCAL TEMPORARY TABLE，greenplum的临时表与Teradata的VOLATILE表同一个意思。
7.UNIQUE表述。
8.不支持MATCH PARTIAL
不支持SQL99的特性，包括：
1.不支持大数据对象CLOB、BLOB和NCLOB,但是其BYTEA和TEXT类型可以保存大数据类型。
2.不支持WITH的递归语句。
3.不支持MODULE。
4.没有CREATE PROCEDURE句法(SQL/PSM)，可以用CREATE FUNCTION代替，返回void，用法如：select my_func(args);
5.greenplum/PostgreSQL的函数定义语言(PL/PGSQL)是Oracle的PL/SQL的子集，不兼容SQL/PSM语言；greenplum还支持用python、perl和R语言实现的函数定义。
6.特意不支持BIT、BIT VARYING类型，此类型在后来的SQL标准中已被取缔。
7.greenplum支持的标识符最长63字符，而SQL标准是128字符。
8.不支持prepared transaction。
9.CHARACTER SET option on the definition of CHAR() or VARCHAR() columns
char和varchar类型对字符集有限制
10.不支持用字符或字节规定char()或varchar()列，比如VARCHAR(15 CHARACTERS) or VARCHAR(15 OCTETS) or VARCHAR(15 BYTES)
11.不支持CURRENT_SCHEMA??????(测试支持，select current_schema())
12.不支持CREATE DISTINCT TYPE语句，CREATE DOMAIN可以作为全能使用。
不支持SQL2003的特性，包括：
1.不支持XML类型，PostgreSQL支持。
2.不支持MERGE语句。
3.不支持IDENTITY列，以及GENERATED ALWAYS/GENERATED BY DEFAULT子句。
4.不支持数据类型的MULTISET修饰符。
5.不支持ROW数据类型。
6.Sequence用法不是标准的，比如nextval(‘seq’)是获取seq的下一个value。
7.不支持GENERATED ALWAYS AS列；可以用视图代替(views can be used as a work-around.)
8.在select的子句中不支持TABLESAMPLE；可以用random()代替。
9.在select和子句中不支持NULLS FIRST或NULLS LAST。
10.在表join时不支持PARTITION BY。
11.不支持对列进行grant select权限操作；可以用view实现。
12.对于CREATE TABLE X (LIKE Y)语句，不支持[INCLUDING|EXCLUDING] [DEFAULTS|CONSTRAINTS|INDEXES] 子句。
13.数组数据类型与SQL标准兼容但有例外，一般用户使用不会受影响。
不支持SQL2008的特性，包括：
1.不支持BINARY和VARBINARY类型，可以用BYTEA类型代替VARBINARY。
2.不支持FETCH FIRST、FETCH NEXT这样的句法，比如select * from sale order by id offset 20 rows fetch next 10 rows only;；可以用limit代替。
3.在VIEW和子查询中，ORDER BY最好与limit一起使用，否则会被忽略，因为可能会导致性能问题。
4.The row subquery construct is not supported.
5.TRUNCATE TABLE不支持CONTINUE IDENTITY和RESTART IDENTITY子句。
6.不支持CREATE FOREIGN DATA WRAPPER (SQL/MED)子句。
7.不支持CREATE SERVER(SQL/MED)子句。
8.不支持CREATE MAPPING(SQL/MED)子句。
参考链接
修改内核参数 (Linux)
Linux内核参数(如kernel.shmmax)及Oracle相关参数调整(如SGA_MAX_SIZE)
How to do failed Segment analysis and recovery
Recovering From Segment Failures
Postgres与OS内核相关的几个参数设置
How to build GPDB on RHEL 7 (如何在RHEL7上编译GPDB)
kenyon的Greenplum的安装