GBASE南大通用8a MPP集群在遇到疑难问题，特别是性能严重异常、现象特别异常时，还有一种关键的故障排查方法，可以跟踪内存的处理日志——memlog；memlog配置方法如下，需要手动配置或下发memlog的配置参数，其中_gbase_inmem_log_level和_gbase_inmem_log_source可以热切，无需重启gbased服务
_gbase_inmem_log_level 参数说明：
0: Fatal
1: Error
2: Warn
3: Info
4: Debug
_gbase_inmem_log_source 参数说明：
1: GENERIC
2: REP
4: CONN
8: SESSION
16: PACKETS
32: EVENTS
64: LOCK
128: CTQ
256: WINS
如果要同时显示多种类型，则set时，可以将数位相加，如显示REP和CONN，则set [global] _gbase_inmem_log_source=6，如果要写入配置文件，则需要在配置文件中，将具体显示类型写入配置文件:_gbase_inmem_log_source=REP,CONN
GBase南大通用8a MPP集群排查问题举例：网络通信异常排查
当trc中出现某个通信阶段消耗的时间较长时，可以通过memlog协助判断gbase 内部通信的问题。
1、memlog打开参数样例
set global _gbase_inmem_log_level=4;
set global _gbase_inmem_log_source=2;
2、memlog使用方法：
1）列出memlog清单
[gbase@localhost ~]$ memlog -l
memlog: find 3 log files, as bellow:
shm_gclusterd_5258
shm_gbased_192.168.80.129_5050
shm_gbased_192.168.80.128_5050
2）对某个实例的memlog进行解析
memlog shm_gbased_192.168.80.129_5050
3、通信的角色和流程
一般内部数据复制（REP）的通信流程：
LINK_INIT ——> OPEN_TABLE ——> GET_TABLE_INFO ——> SendData（发送数据的日志在memlog中不显示） ——> Finish ——> Destroy
4、memlog示例
[gbase@localhost ~]$ memlog shm_gbased_192.168.80.128_5050 【注释】memlog用法
TIME TID LEVEL SOURCE Log
211208 15:39:08 140562791978752 [Info] [REP] GBaseRep.h:0169: Creating Rep instance: 0x5392300?【注释】实例编号
211208 15:39:08 140562791978752 [Info] [REP] GBaseRep.cpp:4644: Rep 0x5392300 opened.
211208 15:39:09 140562790913792 [Info] [REP] GBaseRep.h:0169: Creating Rep instance: 0x5392d00
211208 15:39:09 140562790913792 [Info] [REP] GBaseRep.cpp:4644: Rep 0x5392d00 opened.
211208 15:39:09 140562790913792 [Info] [REP] RepActor.cpp:0131: New REP opened: 0x5392d00, thd: 0x5432000
211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:1179: GBaseRep(0x5392300): role(Initiator), state(Opened), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: LINK_INIT to 192.168.80.128:gctmpdb._tmp_2152769728_7_t1_1_1638949066_s, group_id(0), host_gr?【注释】LINK_INIT 发起通信
211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:1194: GBaseRep(0x5392300): role(Initiator), state(Opened), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sent LINK_INIT to 192.168.80.128:5050, gctmpdb._tmp_2152769728_7_t1_1_1638949066_s
211208 15:39:09 140562790647552 [Info] [REP] GBaseRep.h:0169: Creating Rep instance: 0x5393200
211208 15:39:09 140562790647552 [Info] [REP] GBaseRep.cpp:4644: Rep 0x5393200 opened.
211208 15:39:09 140562790647552 [Info] [REP] RepActor.cpp:0131: New REP opened: 0x5393200, thd: 0x544a000
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5392d00): role(Unknown), state(Opened), scn(-1), src_id() -- Cmd: 0(LINK_INIT) from: 92300 group_id: 0
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:4255: GBaseRep(0x5392d00): role(Unknown), state(Opened), scn(-1), src_id() -- GBaseRep::ReceiveLinkInit <<<<<<<<
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:4271: GBaseRep(0x5392d00): role(Unknown), state(Opened), scn(-1), src_id() -- GBaseRep::SaveTableName <<<<<<<<
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:4342: GBaseRep(0x5392d00): role(Unknown), state(Opened), scn(0), src_id(localhost.localdomain_8_0x5392300) -- ReadRepTable: host(192.168.80.129), table(gctmpdb._tmp_2152769728_7_t1_1_1638949066_s)
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5393200): role(Unknown), state(Opened), scn(-1), src_id() -- Cmd: 0(LINK_INIT) from: e4300 group_id: 0
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4255: GBaseRep(0x5393200): role(Unknown), state(Opened), scn(-1), src_id() -- GBaseRep::ReceiveLinkInit <<<<<<<<
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4271: GBaseRep(0x5393200): role(Unknown), state(Opened), scn(-1), src_id() -- GBaseRep::SaveTableName <<<<<<<<
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4342: GBaseRep(0x5393200): role(Unknown), state(Opened), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- ReadRepTable: host(192.168.80.129), table(gctmpdb._tmp_2152769728_7_t1_1_1638949066_s)
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:1179: GBaseRep(0x5392d00): role(Repeater), state(Opened), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: LINK_INIT to 192.168.80.129:gctmpdb._tmp_2152769728_7_t1_1_1638949066_s, group_id(0), host_gro
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:1194: GBaseRep(0x5392d00): role(Repeater), state(Opened), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sent LINK_INIT to 192.168.80.129:5050, gctmpdb._tmp_2152769728_7_t1_1_1638949066_s
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:4586: GBaseRep(0x5392d00): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: LINK_INIT_ACK to 192.168.80.128, group_id: 0
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1179: GBaseRep(0x5393200): role(Repeater), state(Opened), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: LINK_INIT to 192.168.80.129:gctmpdb._tmp_2152769728_7_t1_1_1638949066_s, group_id(0), host_gro
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1194: GBaseRep(0x5393200): role(Repeater), state(Opened), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sent LINK_INIT to 192.168.80.129:5050, gctmpdb._tmp_2152769728_7_t1_1_1638949066_s
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4586: GBaseRep(0x5393200): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: LINK_INIT_ACK to 192.168.80.129, group_id: 0
211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:3351: GBaseRep(0x5392300): role(Initiator), state(SettingLink), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 1(LINK_INIT_ACK) from: 92d00 group_id: 0?【注释】收到LINK_INIT_ACK
211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:1214: GBaseRep(0x5392300): role(Initiator), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: OPEN_TABLE to 192.168.80.128, group_id: 0?【注释】发起OPEN_TABLE
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5392d00): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 2(OPEN_TABLE) from: 92300 group_id: 0
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4265: GBaseRep(0x5392d00): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- GBaseRep::ReceiveOpenTable <<<<<<<<
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4608: GBaseRep(0x5392d00): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: OPEN_TABLE_ACK to 192.168.80.128, group_id: 0
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1214: GBaseRep(0x5392d00): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: OPEN_TABLE to 192.168.80.129, group_id: 0
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1226: GBaseRep(0x5392d00): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: GET_TABLE_INFO
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1277: GBaseRep(0x5392d00): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: GET_TABLE_INFO to 192.168.80.129:gctmpdb._tmp_2152769728_7_t1_1_1638949066_s, group_id(0)
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1045: GBaseRep(0x5392d00): role(Repeater), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x5392300) -- GBaseRep::PrepareAllReceiveTable, table opened & locked...
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5393200): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Cmd: 2(OPEN_TABLE) from: e4300 group_id: 0
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4265: GBaseRep(0x5393200): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- GBaseRep::ReceiveOpenTable <<<<<<<<
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4608: GBaseRep(0x5393200): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: OPEN_TABLE_ACK to 192.168.80.129, group_id: 0
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1214: GBaseRep(0x5393200): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: OPEN_TABLE to 192.168.80.129, group_id: 0
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1226: GBaseRep(0x5393200): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: GET_TABLE_INFO
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1277: GBaseRep(0x5393200): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: GET_TABLE_INFO to 192.168.80.129:gctmpdb._tmp_2152769728_7_t1_1_1638949066_s, group_id(0)
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1045: GBaseRep(0x5393200): role(Repeater), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- GBaseRep::PrepareAllReceiveTable, table opened & locked...
211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:3351: GBaseRep(0x5392300): role(Initiator), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 3(OPEN_TABLE_ACK) from: 92d00 group_id: 0?【注释】收到OPEN_TABLE_ACK
211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:1226: GBaseRep(0x5392300): role(Initiator), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: GET_TABLE_INFO?【注释】获取表信息
211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:1277: GBaseRep(0x5392300): role(Initiator), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: GET_TABLE_INFO to 192.168.80.128:gctmpdb._tmp_2152769728_7_t1_1_1638949066_s, group_id(0)
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5392d00): role(Repeater), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 4(GET_TABLE_INFO) from: 92300 group_id: 0
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3582: GBaseRep(0x5392d00): role(Repeater), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Skipped Cmd: SISC_GET_TABLE_INFO in state: WaitingTableInfo
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5393200): role(Repeater), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Cmd: 4(GET_TABLE_INFO) from: e4300 group_id: 0
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3582: GBaseRep(0x5393200): role(Repeater), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Skipped Cmd: SISC_GET_TABLE_INFO in state: WaitingTableInfo
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5392d00): role(Repeater), state(TableInfoReplyPending), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 1(LINK_INIT_ACK) from: e4d00 group_id: 0
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5393200): role(Repeater), state(TableInfoReplyPending), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Cmd: 1(LINK_INIT_ACK) from: e5200 group_id: 0
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5392d00): role(Repeater), state(TableInfoReplyPending), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 3(OPEN_TABLE_ACK) from: e4d00 group_id: 0
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5393200): role(Repeater), state(TableInfoReplyPending), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Cmd: 3(OPEN_TABLE_ACK) from: e5200 group_id: 0
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5392d00): role(Repeater), state(TableInfoReplyPending), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 5(BACK_TABLE_INFO) from: e4d00 group_id: 0
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:1314: GBaseRep(0x5392d00): role(Repeater), state(TableInfoReplyPending), scn(0), src_id(localhost.localdomain_8_0x5392300) -- table: 0x3335180, table_info: 0x32a5560
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:4383: GBaseRep(0x5392d00): role(Repeater), state(TableInfoReceived), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: BACK_TABLE_INFO to 192.168.80.128, group_id(0)
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5393200): role(Repeater), state(TableInfoReplyPending), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Cmd: 5(BACK_TABLE_INFO) from: e5200 group_id: 0
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:1314: GBaseRep(0x5393200): role(Repeater), state(TableInfoReplyPending), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- table: 0x3335360, table_info: 0x32a55f0
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:4383: GBaseRep(0x5393200): role(Repeater), state(TableInfoReceived), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: BACK_TABLE_INFO to 192.168.80.129, group_id(0)
211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:3351: GBaseRep(0x5392300): role(Initiator), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 5(BACK_TABLE_INFO) from: 92d00 group_id: 0?【注释】返回表信息
211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:1314: GBaseRep(0x5392300): role(Initiator), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x5392300) -- table: 0x46b4b40, table_info: 0x32a4d80?【注释】收到具体表地址
211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:0507: GBaseRep(0x5392300): role(Initiator), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: FINISH to 192.168.80.128, group_id: -1?【注释】FINISH
211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:0514: GBaseRep(0x5392300): role(Initiator), state(Closing), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Receiving Cmd ACK: FINISH from 192.168.80.128, group_id: -1
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5392d00): role(Repeater), state(TableDataReceived), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 20(FINISH) from: 92300 group_id: -1
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:0507: GBaseRep(0x5392d00): role(Repeater), state(TableDataReceived), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: FINISH to 192.168.80.129, group_id: -1
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5393200): role(Repeater), state(TableDataReceived), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Cmd: 20(FINISH) from: e4300 group_id: -1
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:0507: GBaseRep(0x5393200): role(Repeater), state(TableDataReceived), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: FINISH to 192.168.80.129, group_id: -1
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5392d00): role(Repeater), state(Closing), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 21(FINISH_ACK) from: e4d00 group_id: -1
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3755: GBaseRep(0x5392d00): role(Repeater), state(Closing), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Streams still open: 0
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:4617: GBaseRep(0x5392d00): role(Repeater), state(Closing), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: FINISH_ACK to 192.168.80.128, group_id: -1
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5393200): role(Repeater), state(Closing), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Cmd: 21(FINISH_ACK) from: e5200 group_id: -1
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:3755: GBaseRep(0x5393200): role(Repeater), state(Closing), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Streams still open: 0
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4617: GBaseRep(0x5393200): role(Repeater), state(Closing), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: FINISH_ACK to 192.168.80.129, group_id: -1
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:4699: Closed 0x5392d00, thd: 0x5432000, opened: 0, role: Repeater
211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:5173: Destroying Rep instance: 0x5392d00
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4699: Closed 0x5393200, thd: 0x544a000, opened: 0, role: Repeater
211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:5173: Destroying Rep instance: 0x5393200
211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:3351: GBaseRep(0x5392300): role(Initiator), state(Closing), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 21(FINISH_ACK) from: 92d00 group_id: -1?【注释】收到FINISH_ACK
211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:4699: Closed 0x5392300, thd: 0x5378000, opened: 0, role: Initiator
211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:5173: Destroying Rep instance: 0x5392300?【注释】销毁通信实例
Total logs: 78
以上示例记录了正常数据复制的详细记录，当数据复制的过程出现问题时，可以通过menlong做一个深入诊断，判断当前处理机制是否存在问题；
memlog可以显示多种类型的日志，比如CONN、SESSION、PACKETS等，可以通过配置参数进行设置，方便用户分析排查；
最后，memlog需要结合专业的产品知识做为基础，才能帮助排查分析问题，在一定条件下，也可以通过该手段协助专业工程师进行信息收集，提高排查问题的效率。