Linux创建macvlan,测试bridge、private和vepa模式
最近在看Docker的网络,看到关于macvlan网络的介绍。查阅了相关资料,记录如下。
1.Linux Macvlan
2.图解几个与Linux网络虚拟化相关的虚拟网卡-VETH/MACVLAN/MACVTAP/IPVLAN
Centos7.9
本章节内容引自参考链接2.图解几个与Linux网络虚拟化相关的虚拟网卡-VETH/MACVLAN/MACVTAP/IPVLAN
MACVLAN技术可谓是提出一种将一块以太网卡虚拟成多块以太网卡的极简单的方案。一块以太网卡需要有一个MAC地址,这就是以太网卡的核心中的核心。
以往,我们只能为一块以太网卡添加多个IP地址,却不能添加多个MAC地址,因为MAC地址正是通过其全球唯一性来标识一块以太网卡的,即便你使用了创建ethx:y这样的方式,你会发现所有这些“网卡”的MAC地址和ethx都是一样的,本质上,它们还是一块网卡,这将限制你做很多二层的操作。有了MACVLAN技术,你可以这么做了。
我们先来看一下MACVLAN技术的流程示意图:
在具体的执行上,通过下面的命令,你可以创建一个MACVLAN网卡,它是基于eth0虚拟出来的:
ip link add link eth0 name macv1 type macvlan
你可以认为有人将双绞线“物理上”每根一分为二,接了两个水晶头,从而连接了两块网卡,其中一块是虚拟的MACVLAN网卡。但是既然共享介质,难道不用运行CSMA/CD吗?当然不用,因为事实上,最终的数据是通过eth0发出的,而现代的以太网卡工作的全双工模式,只要是交换式全双工(某些标准而言,这是必须的),eth0自己能做好。
现在可以说一下MACVLAN技术构建的虚拟网卡的模式了。之所以MACVLAN拥有所谓的模式,是因为相比VETH,它更是将复杂性建立在了一个已经容不下什么的以太网概念上,因此相互交互的元素就会太多,它们之间的关系不同,导致最终MACVLAN的行为不同。还是图解的方式:
这个bridge只是针对同属于一块宿主以太网卡的MACVLAN网卡以及宿主网卡之间的通信行为的,与外部通信无关。所谓的bridge指的是在这些网卡之间,数据流可以实现直接转发,不需要外部的协助,这有点类似于Linux BOX内建了一个bridge,即用brctl命令所做的那一切。
VEPA模式我后面会专门讲。现在要知道的是,在VEPA模式下,即使是MACVLANeth1和MACVLANeth2同时配在在eth0上,它们两者之间的通信也不能直接进行,而必须通过与eth0相连的外部的交换机协助,这通常是一个支持“发夹弯”转发的交换机。
这种private模式的隔离强度比VEPA更强。在private模式下,即使是MACVLANeth1和MACVLANeth2同时配在在eth0上,eth0连接了外部交换机S,S支持“发夹弯”转发模式,即便这样,MACVLANeth1的广播/多播流量也无法到达MACVLANeth2,反之亦然,之所以隔离广播流量,是因为以太网是基于广播的,隔离了广播,以太网将失去了依托。
通过Linux 命令创建两个macvlan子网卡,分别加入两个网络命名空间(namespace),测试同一父网卡的两个子macvlan网卡在bridge、private和vepa模式下的网络通讯。
安装测试环境需要的包
[root@centos7-10 ~]# yum install -y net-tools iputils telnet traceroute iproute bridge-utils NetworkManager
如果是ubuntu 命令如下:
apt install -y net-tools inetutils-ping telnet traceroute iproute2 bridge-utils network-manager
macvlan网络需要打开网卡的混杂模式
[root@centos7-18 ~]# ip link set enp0s5 promisc on
[root@centos7-18 ~]#
[root@centos7-18 ~]# ip a | grep "enp0s5"
2: enp0s5: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 10.211.55.18/24 brd 10.211.55.255 scope global enp0s5
创建两个网络命名空间(namespace)用于隔离两块macvlan子网卡
// 创建namespace ns1和ns2
[root@centos7-18 ~]# ip netns add ns1
[root@centos7-18 ~]# ip netns add ns2
[root@centos7-18 ~]#
[root@centos7-18 ~]# ip netns list
ns2
ns1
[root@centos7-18 ~]#
测试macvlan的bridge、private和vepa模式。
// 创建两个macvlan网卡,模式bridge
[root@centos7-18 ~]# ip link add link enp0s5 name enp0s5.101 type macvlan mode bridge
[root@centos7-18 ~]# ip link add link enp0s5 name enp0s5.102 type macvlan mode bridge
[root@centos7-18 ~]# ip a | grep -A2 "enp0s5\."
8: enp0s5.101@enp0s5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether ba:9e:8b:60:84:ea brd ff:ff:ff:ff:ff:ff
9: enp0s5.102@enp0s5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 1a:97:08:af:84:b2 brd ff:ff:ff:ff:ff:ff
[root@centos7-18 ~]# ip link set enp0s5.101 netns ns1
[root@centos7-18 ~]#
[root@centos7-18 ~]# ip link set enp0s5.102 netns ns2
[root@centos7-18 ~]#
// 配置网卡IP 10.211.55.101
[root@centos7-18 ~]# ip netns exec ns1 ip address add 10.211.55.101/24 dev enp0s5.101
// 配置网卡混杂模式 promisc
[root@centos7-18 ~]# ip netns exec ns1 ip link set enp0s5.101 promisc on
// 配置网卡 up
[root@centos7-18 ~]# ip netns exec ns1 ip link set enp0s5.101 up
[root@centos7-18 ~]#
// 查看网卡信息
[root@centos7-18 ~]# ip netns exec ns1 ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
12: enp0s5.101@if2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether c6:83:c9:d6:6a:b4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.211.55.101/24 scope global enp0s5.101
valid_lft forever preferred_lft forever
inet6 fe80::c483:c9ff:fed6:6ab4/64 scope link
valid_lft forever preferred_lft forever
// 配置网卡IP 10.211.55.102
[root@centos7-18 ~]# ip netns exec ns2 ip address add 10.211.55.102/24 dev enp0s5.102
// 配置网卡混杂模式 promisc
[root@centos7-18 ~]# ip netns exec ns2 ip link set enp0s5.102 promisc on
// 配置网卡 up
[root@centos7-18 ~]# ip netns exec ns2 ip link set enp0s5.102 up
[root@centos7-18 ~]#
// 查看网卡信息
[root@centos7-18 ~]# ip netns exec ns2 ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
13: enp0s5.102@if2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:c6:89:d2:87:73 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.211.55.102/24 scope global enp0s5.102
valid_lft forever preferred_lft forever
inet6 fdb2:2c26:f4e4:0:50c6:89ff:fed2:8773/64 scope global mngtmpaddr dynamic
valid_lft 2591994sec preferred_lft 604794sec
inet6 fe80::50c6:89ff:fed2:8773/64 scope link
valid_lft forever preferred_lft forever
[root@centos7-18 ~]#
[root@centos7-18 ~]# ip netns exec ns1 ping -c2 10.211.55.102
PING 10.211.55.102 (10.211.55.102) 56(84) bytes of data.
64 bytes from 10.211.55.102: icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from 10.211.55.102: icmp_seq=2 ttl=64 time=0.066 ms
--- 10.211.55.102 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.045/0.055/0.066/0.012 ms
[root@centos7-18 ~]# ip netns exec ns1 ping -c2 10.211.55.1
PING 10.211.55.1 (10.211.55.1) 56(84) bytes of data.
64 bytes from 10.211.55.1: icmp_seq=1 ttl=128 time=0.156 ms
64 bytes from 10.211.55.1: icmp_seq=2 ttl=128 time=0.181 ms
--- 10.211.55.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.156/0.168/0.181/0.018 ms
[root@centos7-18 ~]# ip netns exec ns1 ping -c2 10.211.55.10
PING 10.211.55.10 (10.211.55.10) 56(84) bytes of data.
64 bytes from 10.211.55.10: icmp_seq=1 ttl=64 time=0.276 ms
64 bytes from 10.211.55.10: icmp_seq=2 ttl=64 time=0.509 ms
--- 10.211.55.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.276/0.392/0.509/0.118 ms
[root@centos7-18 ~]# ip netns exec ns1 ping -c2 10.211.55.18
PING 10.211.55.18 (10.211.55.18) 56(84) bytes of data.
--- 10.211.55.18 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms
// 删掉之前的配置,以免影响测试
[root@centos7-18 ~]# ip netns exec ns1 ip link del enp0s5.101
[root@centos7-18 ~]# ip netns exec ns2 ip link del enp0s5.102
// 创建两个macvlan网卡,模式private
[root@centos7-18 ~]# ip link add link enp0s5 name enp0s5.111 type macvlan mode private
[root@centos7-18 ~]# ip link add link enp0s5 name enp0s5.112 type macvlan mode private
[root@centos7-18 ~]# ip a | grep -A2 "enp0s5\."
16: enp0s5.111@enp0s5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether ae:7d:a9:97:13:e1 brd ff:ff:ff:ff:ff:ff
17: enp0s5.112@enp0s5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether ee:2e:82:69:b5:66 brd ff:ff:ff:ff:ff:ff
[root@centos7-18 ~]#
[root@centos7-18 ~]# ip link set enp0s5.111 netns ns1
[root@centos7-18 ~]#
[root@centos7-18 ~]# ip link set enp0s5.112 netns ns2
[root@centos7-18 ~]#
// 配置网卡IP 10.211.55.111
[root@centos7-18 ~]# ip netns exec ns1 ip address add 10.211.55.111/24 dev enp0s5.111
// 配置网卡混杂模式 promisc
[root@centos7-18 ~]# ip netns exec ns1 ip link set enp0s5.111 promisc on
// 配置网卡 up
[root@centos7-18 ~]# ip netns exec ns1 ip link set enp0s5.111 up
[root@centos7-18 ~]#
// 查看网卡信息
[root@centos7-18 ~]# ip netns exec ns1 ip a | grep -A5 enp0s5.111
16: enp0s5.111@if2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ae:7d:a9:97:13:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.211.55.111/24 scope global enp0s5.111
valid_lft forever preferred_lft forever
inet6 fdb2:2c26:f4e4:0:ac7d:a9ff:fe97:13e1/64 scope global mngtmpaddr dynamic
valid_lft 2591930sec preferred_lft 604730sec
inet6 fe80::ac7d:a9ff:fe97:13e1/64 scope link
valid_lft forever preferred_lft forever
[root@centos7-18 ~]#
// 配置网卡IP 10.211.55.112
[root@centos7-18 ~]# ip netns exec ns2 ip address add 10.211.55.112/24 dev enp0s5.112
// 配置网卡混杂模式 promisc
[root@centos7-18 ~]# ip netns exec ns2 ip link set enp0s5.112 promisc on
// 配置网卡 up
[root@centos7-18 ~]# ip netns exec ns2 ip link set enp0s5.112 up
[root@centos7-18 ~]#
// 查看网卡信息
[root@centos7-18 ~]# ip netns exec ns2 ip a | grep -A5 enp0s5.112
17: enp0s5.112@if2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ee:2e:82:69:b5:66 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.211.55.112/24 scope global enp0s5.112
valid_lft forever preferred_lft forever
inet6 fdb2:2c26:f4e4:0:ec2e:82ff:fe69:b566/64 scope global mngtmpaddr dynamic
valid_lft 2591822sec preferred_lft 604622sec
inet6 fe80::ec2e:82ff:fe69:b566/64 scope link
valid_lft forever preferred_lft forever
[root@centos7-18 ~]#
// Ping ns1下10.211.55.111,不通
[root@centos7-18 ~]# ip netns exec ns2 ping -c2 10.211.55.111
PING 10.211.55.111 (10.211.55.111) 56(84) bytes of data.
--- 10.211.55.111 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1000ms
// Ping 网络网关10.211.55.1,通
[root@centos7-18 ~]# ip netns exec ns2 ping -c2 10.211.55.1
PING 10.211.55.1 (10.211.55.1) 56(84) bytes of data.
64 bytes from 10.211.55.1: icmp_seq=1 ttl=128 time=0.165 ms
64 bytes from 10.211.55.1: icmp_seq=2 ttl=128 time=0.315 ms
--- 10.211.55.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.165/0.240/0.315/0.075 ms
// Ping 网络其他IP 10.211.55.10,通
[root@centos7-18 ~]# ip netns exec ns2 ping -c2 10.211.55.10
PING 10.211.55.10 (10.211.55.10) 56(84) bytes of data.
64 bytes from 10.211.55.10: icmp_seq=1 ttl=64 time=0.289 ms
64 bytes from 10.211.55.10: icmp_seq=2 ttl=64 time=0.525 ms
--- 10.211.55.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.289/0.407/0.525/0.118 ms
// Ping 父网卡10.211.55.18,不通
[root@centos7-18 ~]# ip netns exec ns2 ping -c2 10.211.55.18
PING 10.211.55.18 (10.211.55.18) 56(84) bytes of data.
--- 10.211.55.18 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1000ms
[root@centos7-18 ~]#
// 删掉之前的配置,以免影响测试
[root@centos7-18 ~]# ip netns exec ns1 ip link del enp0s5.111
[root@centos7-18 ~]# ip netns exec ns2 ip link del enp0s5.112
// 创建两个macvlan网卡,模式vepa
[root@centos7-18 ~]# ip link add link enp0s5 name enp0s5.121 type macvlan mode vepa
[root@centos7-18 ~]# ip link add link enp0s5 name enp0s5.122 type macvlan mode vepa
[root@centos7-18 ~]# ip a | grep -A5 "enp0s5\."
18: enp0s5.121@enp0s5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 5a:3a:26:f1:72:c5 brd ff:ff:ff:ff:ff:ff
19: enp0s5.122@enp0s5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether f6:16:e0:70:79:99 brd ff:ff:ff:ff:ff:ff
[root@centos7-18 ~]#
[root@centos7-18 ~]# ip link set enp0s5.121 netns ns1
[root@centos7-18 ~]# ip link set enp0s5.122 netns ns2
[root@centos7-18 ~]#
// 配置网卡IP 10.211.55.121
[root@centos7-18 ~]# ip netns exec ns1 ip address add 10.211.55.121/24 dev enp0s5.121
// 配置网卡混杂模式 promisc
[root@centos7-18 ~]# ip netns exec ns1 ip link set enp0s5.121 promisc on
// 配置网卡 up
[root@centos7-18 ~]# ip netns exec ns1 ip link set enp0s5.121 up
[root@centos7-18 ~]#
// 查看网卡信息
[root@centos7-18 ~]# ip netns exec ns1 ip a | grep -A5 enp0s5.121
18: enp0s5.121@if2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 5a:3a:26:f1:72:c5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.211.55.121/24 scope global enp0s5.121
valid_lft forever preferred_lft forever
inet6 fdb2:2c26:f4e4:0:583a:26ff:fef1:72c5/64 scope global mngtmpaddr dynamic
valid_lft 2591839sec preferred_lft 604639sec
inet6 fe80::583a:26ff:fef1:72c5/64 scope link
valid_lft forever preferred_lft forever
[root@centos7-18 ~]#
// 配置网卡IP 10.211.55.112
[root@centos7-18 ~]# ip netns exec ns2 ip address add 10.211.55.122/24 dev enp0s5.122
// 配置网卡混杂模式 promisc
[root@centos7-18 ~]# ip netns exec ns2 ip link set enp0s5.122 promisc on
// 配置网卡 up
[root@centos7-18 ~]# ip netns exec ns2 ip link set enp0s5.122 up
[root@centos7-18 ~]#
// 查看网卡信息
[root@centos7-18 ~]# ip netns exec ns2 ip a | grep -A5 enp0s5.122
19: enp0s5.122@if2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether f6:16:e0:70:79:99 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.211.55.122/24 scope global enp0s5.122
valid_lft forever preferred_lft forever
inet6 fdb2:2c26:f4e4:0:f416:e0ff:fe70:7999/64 scope global mngtmpaddr dynamic
valid_lft 2591925sec preferred_lft 604725sec
inet6 fe80::f416:e0ff:fe70:7999/64 scope link
valid_lft forever preferred_lft forever
[root@centos7-18 ~]#
// Ping ns1下10.211.55.121,
// 原理是可以通,但我的网卡或交换不支持802.1q协议,造成测试不通
[root@centos7-18 ~]# ip netns exec ns2 ping -c2 10.211.55.121
PING 10.211.55.121 (10.211.55.121) 56(84) bytes of data.
--- 10.211.55.121 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1000ms
// Ping 网络网关10.211.55.1,通
[root@centos7-18 ~]# ip netns exec ns2 ping -c2 10.211.55.1
PING 10.211.55.1 (10.211.55.1) 56(84) bytes of data.
64 bytes from 10.211.55.1: icmp_seq=1 ttl=128 time=0.280 ms
64 bytes from 10.211.55.1: icmp_seq=2 ttl=128 time=0.184 ms
--- 10.211.55.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.184/0.232/0.280/0.048 ms
// Ping 网络其他IP 10.211.55.10,通
[root@centos7-18 ~]# ip netns exec ns2 ping -c2 10.211.55.10
PING 10.211.55.10 (10.211.55.10) 56(84) bytes of data.
64 bytes from 10.211.55.10: icmp_seq=1 ttl=64 time=0.504 ms
64 bytes from 10.211.55.10: icmp_seq=2 ttl=64 time=0.507 ms
--- 10.211.55.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.504/0.505/0.507/0.022 ms
// Ping 父网卡10.211.55.18,不通
[root@centos7-18 ~]# ip netns exec ns2 ping -c2 10.211.55.18
PING 10.211.55.18 (10.211.55.18) 56(84) bytes of data.
--- 10.211.55.18 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms
[root@centos7-18 ~]#
因为交换不支持802.1q,可以采用Linux bridge开启hairpin方式,模拟测试macvlan vepa模式,方法详见我的另一篇文章,链接如下: