{SOLVED} корады ищу советов по IB

Пытаюсь соединить 3 пк в кольцо
1 пк -
ifconfig

ib0: flags=4163  mtu 2044
        inet 10.0.0.1  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::202:c903:f:6d27  prefixlen 64  scopeid 0x20
        Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
        infiniband 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  txqueuelen 256  (InfiniBand)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 7 overruns 0  carrier 0  collisions 0

ib1: flags=4163  mtu 2044
        inet 10.0.0.2  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::202:c903:f:6d28  prefixlen 64  scopeid 0x20
        Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
        infiniband 80:00:00:49:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  txqueuelen 256  (InfiniBand)
        RX packets 12  bytes 696 (696.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 7 overruns 0  carrier 0  collisions 0

ibstat
CA 'mlx4_0'
	CA type: MT26428
	Number of ports: 2
	Firmware version: 2.9.1000
	Hardware version: a0
	Node GUID: 0x0002c903000f6d26
	System image GUID: 0x0002c903000f6d29
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 40
		Base lid: 2
		LMC: 0
		SM lid: 1
		Capability mask: 0x0259086a
		Port GUID: 0x0002c903000f6d27
		Link layer: InfiniBand
	Port 2:
		State: Active
		Physical state: LinkUp
		Rate: 40
		Base lid: 2
		LMC: 0
		SM lid: 5
		Capability mask: 0x02590868
		Port GUID: 0x0002c903000f6d28
		Link layer: InfiniBand

2 ПК -

1. ip 10.0.0.3 10.0.0.4
2. netstat
CA 'mlx4_0'
	CA type: MT26428
	Number of ports: 2
	Firmware version: 2.9.1000
	Hardware version: a0
	Node GUID: 0x0002c90300048974
	System image GUID: 0x0002c90300048977
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 40
		Base lid: 4
		LMC: 0
		SM lid: 4
		Capability mask: 0x0259086a
		Port GUID: 0x0002c90300048975
		Link layer: InfiniBand
	Port 2:
		State: Active
		Physical state: LinkUp
		Rate: 40
		Base lid: 1
		LMC: 0
		SM lid: 1
		Capability mask: 0x0259086a
		Port GUID: 0x0002c90300048976
		Link layer: InfiniBand

3 ПК

1 ip 10.0.0.5 10.0.0.6
2 ibstat
CA 'mlx4_0'
	CA type: MT26428
	Number of ports: 2
	Firmware version: 2.9.1000
	Hardware version: a0
	Node GUID: 0x0002c9030004a772
	System image GUID: 0x0002c9030004a775
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 40
		Base lid: 3
		LMC: 0
		SM lid: 4
		Capability mask: 0x0259086a
		Port GUID: 0x0002c9030004a773
		Link layer: InfiniBand
	Port 2:
		State: Active
		Physical state: LinkUp
		Rate: 40
		Base lid: 5
		LMC: 0
		SM lid: 5
		Capability mask: 0x0259086a
		Port GUID: 0x0002c9030004a774
		Link layer: InfiniBand

узел 1 видит
ibnodes

Ca	: 0x0002c90300048974 ports 2 "filo_server2 HCA-1"
Ca	: 0x0002c903000f6d26 ports 2 "filo_server1 HCA-1"

узел 2

ibnodes 
Ca	: 0x0002c9030004a772 ports 2 "filo_server3 HCA-1"
Ca	: 0x0002c90300048974 ports 2 "filo_server2 HCA-1"

узел 3

Ca	: 0x0002c90300048974 ports 2 "filo_server2 HCA-1"
Ca	: 0x0002c9030004a772 ports 2 "filo_server3 HCA-1"

Скорость между узлами iperf3 -c 10.0.0.3

Connecting to host 10.0.0.3, port 5201
[  4] local 10.0.0.5 port 54500 connected to 10.0.0.3 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   779 MBytes  6.54 Gbits/sec    0   1.57 MBytes       
[  4]   1.00-2.00   sec   440 MBytes  3.69 Gbits/sec    0   1.57 MBytes       
[  4]   2.00-3.00   sec   465 MBytes  3.90 Gbits/sec    0   1.57 MBytes       
[  4]   3.00-4.00   sec   539 MBytes  4.52 Gbits/sec    0   1.57 MBytes       
[  4]   4.00-5.00   sec   634 MBytes  5.32 Gbits/sec    0   1.57 MBytes       
[  4]   5.00-6.00   sec   629 MBytes  5.27 Gbits/sec    0   1.57 MBytes       
[  4]   6.00-7.00   sec   654 MBytes  5.48 Gbits/sec    0   1.57 MBytes       
[  4]   7.00-8.00   sec   690 MBytes  5.79 Gbits/sec    0   1.57 MBytes       
[  4]   8.00-9.00   sec   769 MBytes  6.45 Gbits/sec    0   1.57 MBytes       
[  4]   9.00-10.00  sec   638 MBytes  5.35 Gbits/sec    0   1.57 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  6.09 GBytes  5.23 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  6.09 GBytes  5.23 Gbits/sec                  receiver

sminfo

1 sminfo: sm lid 1 sm guid 0x2c90300048976, activity count 198347 priority 0 state 3 SMINFO_MASTER
2 sminfo: sm lid 4 sm guid 0x2c90300048975, activity count 153356 priority 0 state 3 SMINFO_MASTER
3 sminfo: sm lid 4 sm guid 0x2c90300048975, activity count 153368 priority 0 state 3 SMINFO_MASTER

Вопросы
1. Где сменить умена узлов (HCA-...) на свои.
2. узел 1 видит узел 2 через первый порт, но не видит узел 3 через второй порт. (соединение кольцо) Влияют ли разные маски? если да то где их сменить?
3. как понять вывод sminfo?
4. Скорость такая низкая из-за datagram mode? (l2 mtu 2044)
5. Установлен OFED c репозитория в мане к OpenSM стоит что нужно OpenIB. но его не поствить из-за зависимостей с OFED. В Ofed есть флаг +opensm и
filo_server1 opensm # eix ofed
[I] sys-infiniband/ofed
Available versions:
(3.5) (~)3.5
(3.12) (~)3.12_rc1
{compat-dapl dapl +diags ibacm mstflint +opensm perftest qperf rds sdp srp OFED_DRIVERS="cxgb3 cxgb4 ehca ipath mlx4 mlx5 mthca nes ocrdma psm"}
Installed versions: 3.12_rc1(3.12)(10:39:11 12.06.2015)(dapl diags ibacm mstflint opensm perftest qperf -rds -srp OFED_DRIVERS="mlx4 -cxgb3 -cxgb4 -ehca -ipath -mlx5 -mthca -nes -ocrdma -psm")
Homepage: http://www.openfabrics.org/
Description: OpenIB system files
т.е. ставить OpenIB не надо?

Где почитать про использование IB? на mellanox все прочитал но у меня установленно из гентушных репов OFED в не с сайта изготовителя