Calico BGP 网络（v2.6.x）

搜索文档…
基于版本为 Calico v2.6.x「当前官方最新版本为 v3.0 Calico Reference」
calicoctl
calicoctl 是 calico 网络命令行管理工具。
概览
1
# calicoctl --help
2
Usage:
3
  calicoctl [options] <command> [<args>...]
4
​
5
    create    Create a resource by filename or stdin.   
6
              // 从标准输入或者文件创建资源
7
    replace   Replace a resource by filename or stdin.  
8
              // 从标准输入或者文件更新资源
9
    apply     Apply a resource by filename or stdin.  This creates a resource
10
              if it does not exist, and replaces a resource if it does exists.
11
              // 从标准输入或者文件应用资源，如果资源存在则更新「Replace」，不在则创建 「Create」
12
    delete    Delete a resource identified by file, stdin or resource type and
13
              name.
14
              // 从文件、标准输出或者资源类型和名字删除资源
15
    get       Get a resource identified by file, stdin or resource type and
16
              name.
17
              // 通过文件、标准输入或者资源类型和名字获取定义的资源
18
    config    Manage system-wide and low-level node configuration options.
19
              // 管理系统层和较低级别的节点配置选项
20
    ipam      IP address management.
21
              // IP 地址管理
22
    node      Calico node management.
23
              // Calico 节点管理
24
    version   Display the version of calicoctl.
25
              // 显示 calicoctl 版本
26
​
27
Options:
28
  -h --help               Show this screen.
29
  -l --log-level=<level>  Set the log level (one of panic, fatal, error,
30
                          warn, info, debug) [default: panic]
31
                          // 设置日志级别
Copied!
create
1
# calicoctl create --help
2
Set the Calico datastore access information in the environment variables or
3
or supply details in a config file.
4
​
5
Usage:
6
  calicoctl create --filename=<FILENAME> [--skip-exists] [--config=<CONFIG>]
7
​
8
Examples:
9
  # Create a policy using the data in policy.yaml.
10
  # 通过 yaml 文件创建对应资源
11
  calicoctl create -f ./policy.yaml
12
​
13
  # Create a policy based on the JSON passed into stdin.
14
  # 通过传递 json 内容到标准输出创建对应资源
15
  cat policy.json | calicoctl create -f -
16
​
17
Options:
18
  -h --help                 Show this screen.
19
  -f --filename=<FILENAME>  Filename to use to create the resource.  If set to
20
                            "-" loads from stdin.
21
     --skip-exists          Skip over and treat as successful any attempts to
22
                            create an entry that already exists.
23
  -c --config=<CONFIG>      Path to the file containing connection
24
                            configuration in YAML or JSON format.
25
                            [default: /etc/calico/calicoctl.cfg]
26
... ...
27
Valid resource types are:
28
​
29
  * node
30
  * bgpPeer
31
  * hostEndpoint
32
  * workloadEndpoint
33
  * ipPool
34
  * policy
35
  * profile
36
... ...
Copied!
v3.0 中新增了 -n --namespace=<NS> 选项
replace
使用选项同 create，replace 用于更新，如果资源对象不存在则抛错。
apply
使用选项同 create，apply 执行时如果资源不存在则创建该资源对象，如果存在则更新。
delete
1
# calicoctl delete --help
2
Set the Calico datastore access information in the environment variables or
3
or supply details in a config file.
4
​
5
Usage:
6
  calicoctl delete ([--scope=<SCOPE>] [--node=<NODE>] [--orchestrator=<ORCH>]
7
                    [--workload=<WORKLOAD>] (<KIND> [<NAME>]) |
8
                   --filename=<FILE>)
9
                   [--skip-not-exists] [--config=<CONFIG>]
10
​
11
Examples:
12
  # Delete a policy using the type and name specified in policy.yaml.
13
  calicoctl delete -f ./policy.yaml
14
​
15
  # Delete a policy based on the type and name in the YAML passed into stdin.
16
  cat policy.yaml | calicoctl delete -f -
17
​
18
  # Delete policy with name "foo"
19
  calicoctl delete policy foo
20
​
21
Options:
22
  -h --help                 Show this screen.
23
  -s --skip-not-exists      Skip over and treat as successful, resources that
24
                            don't exist.
25
  -f --filename=<FILENAME>  Filename to use to delete the resource.  If set to
26
                            "-" loads from stdin.
27
  -n --node=<NODE>          The node (this may be the hostname of the compute
28
                            server if your installation does not explicitly set
29
                            the names of each Calico node).
30
     --orchestrator=<ORCH>  The orchestrator (valid for workload endpoints).
31
     --workload=<WORKLOAD>  The workload (valid for workload endpoints).
32
     --scope=<SCOPE>        The scope of the resource type.  One of global,
33
                            node.  This is only valid for BGP peers and is used
34
                            to indicate whether the peer is a global peer or
35
                            node-specific.
36
  -c --config=<CONFIG>      Path to the file containing connection
37
                            configuration in YAML or JSON format.
38
                            [default: /etc/calico/calicoctl.cfg]
39
... ...
40
Valid resource types are:
41
​
42
  * node
43
  * bgpPeer
44
  * hostEndpoint
45
  * workloadEndpoint
46
  * ipPool
47
  * policy
48
  * profile
49
... ...
Copied!
get
1
# List all policy in default output format.
2
 calicoctl get policy
3
​
4
# List a specific policy in YAML format
5
calicoctl get -o yaml policy my-policy-1
Copied!
1
-o --output=<OUTPUT FORMAT>  Output format.  One of: yaml, json, ps, wide,
2
                             custom-columns=..., go-template=...,
3
                             go-template-file=...   [Default: ps]
Copied!
默认 get 命令输出格式为 ps
1
$ calicoctl get hostEndpoint
2
HOSTNAME   NAME        
3
host1      endpoint1   
4
myhost     eth0
Copied!
wide 格式输出会更详细，会输出资源的一些附加列
1
$ calicoctl get hostEndpoint --output=wide
2
HOSTNAME   NAME        INTERFACE   IPS                PROFILES      
3
host1      endpoint1               1.2.3.4,0:bb::aa   prof1,prof2   
4
myhost     eth0                                       profile1
Copied!
custom-columns 可以自定义输出列
1
$ calicoctl get hostEndpoint --output=custom-columns=NAME,IPS
2
NAME        IPS                
3
endpoint1   1.2.3.4,0:bb::aa   
4
eth0
Copied!
yaml/json 以 yaml 或者 json 格式输出
1
$ calicoctl get hostEndpoint --output=yaml
2
- apiVersion: v1
3
  kind: hostEndpoint
4
  metadata:
5
    hostname: host1
6
    labels:
7
      type: database
8
    name: endpoint1
9
  spec:
10
    expectedIPs:
11
    - 1.2.3.4
12
    - 0:bb::aa
13
... ...
Copied!
如果节点没有运行 etcd，那么需要通过 ETCD_ENDPOINTS 指定 etcd 地址，否则将无法操作：
1
ETCD_ENDPOINTS=http://172.16.0.10:2379 calicoctl get bgppeers
Copied!
config
1
# calicoctl config --help
2
Set the Calico datastore access information in the environment variables or
3
or supply details in a config file.
4
​
5
Usage:
6
  calicoctl config set <NAME> <VALUE> [--node=<NODE>]
7
                                      [--raw=(bgp|felix)]
8
                                      [--config=<CONFIG>]
9
  calicoctl config unset <NAME> [--node=<NODE>]
10
                                [--raw=(bgp|felix)]
11
                                [--config=<CONFIG>]
12
  calicoctl config get <NAME> [--node=<NODE>]
13
                              [--raw=(bgp|felix)]
14
                              [--config=<CONFIG>]
15
​
16
Examples:
17
  # Turn off the full BGP node-to-node mesh
18
  calicoctl config set nodeToNodeMesh off
19
​
20
  # Set global log level to warning
21
  calicoctl config set logLevel warning
22
​
23
  # Set log level to info for node "node1"
24
  calicoctl config set logLevel info --node=node1
25
​
26
  # Display the current setting for the nodeToNodeMesh
27
  calicoctl config get nodeToNodeMesh
28
​
29
Options:
30
  -n --node=<NODE>      The node name.
31
     --raw=(bgp|felix)  Apply raw configuration for the specified component.
32
                        This option should be used with care; the data is not
33
                        validated and it is possible to configure or remove
34
                        data that may prevent the component from working as
35
                        expected.
36
  -c --config=<CONFIG>  Path to the file containing connection configuration in
37
                        YAML or JSON format.
38
                        [default: /etc/calico/calicoctl.cfg]
39
​
40
... ...
41
​
42
 Name            | Scope       | Value                                  |
43
-----------------+-------------+----------------------------------------+
44
 logLevel        | global,node | none,debug,info,warning,error,critical |
45
 nodeToNodeMesh  | global      | on,off                                 |
46
 asNumber        | global      | 0-4294967295                           |
47
 ipip            | global      | on,off                                 |
Copied!
目前 calicoctl config 只有 logLevel 可以单独设置节点，其它如 nodeToNodeMesh、asNumber、ipip 配置的都是全局选项。默认安装之后 nodeToNodeMesh 为开启状态，如果需要和内部交换机打通，需要通过如下命令关闭该选项：
1
# calicoctl config get nodeToNodeMesh       // 获取当前 nodeToNodeMesh 值，显示为 on
2
on
3
# calicoctl config set nodeToNodeMesh off   // 关闭 nodeToNodeMesh
Copied!
ipam
1
Usage:
2
  calicoctl ipam <command> [<args>...]
3
​
4
    release      Release a Calico assigned IP address.
5
    show         Show details of a Calico assigned IP address.
6
​
7
Options:
8
  -h --help      Show this screen.
9
​
10
Description:
11
  IP Address Management specific commands for calicoctl.
12
​
13
  See 'calicoctl ipam <command> --help' to read about a specific subcommand.
Copied!
目前 calicoctl ipam 的地址管理相对 v2.0 以下的版本，功能还是比较弱的，有 release 和 show 两个命令。
calico ipam release 用于从 Calico 清除未被正常回收的地址
1
$ calicoctl ipam release --ip=192.168.1.2
Copied!
calico ipam show 用于获取指定 ip 地址使用情况
1
# IP is not assigned to an endpoint
2
$ calicoctl ipam show --ip=192.168.1.2
3
IP 192.168.1.2 is not currently assigned
4
​
5
# Basic Docker container has the assigned IP
6
# 表明该 IP 地址已绑定 Docker 容器
7
$ calicoctl ipam show --ip=192.168.1.1
8
No attributes defined for 192.168.1.1
Copied!
node
1
Usage:
2
  calicoctl node <command> [<args>...]
3
​
4
    status       View the current status of a Calico node.
5
                 // 获取 Calico 节点当前状态
6
    diags        Gather a diagnostics bundle for a Calico node.
7
                 // 收集节点诊断信息
8
    checksystem  Verify the compute host is able to run a Calico node instance.
9
                 // 验证系统环境是否可以运行 Calico 节点实例
10
​
11
Options:
12
  -h --help      Show this screen.
13
​
14
Description:
15
  Node specific commands for calicoctl.  These commands must be run directly on
16
  the compute host running the Calico node instance.
17
​
18
  See 'calicoctl node <command> --help' to read about a specific subcommand.
Copied!
1
# calicoctl node --help
2
Set the Calico datastore access information in the environment variables or
3
or supply details in a config file.
4
​
5
Usage:
6
  calicoctl node <command> [<args>...]
7
​
8
    run          Run the Calico node container image.
9
                // 运行节点容器镜像
10
    status       View the current status of a Calico node.
11
                // 获取 Calico 节点当前状态
12
    diags        Gather a diagnostics bundle for a Calico node.
13
                // 收集节点诊断信息
14
    checksystem  Verify the compute host is able to run a Calico node instance.
15
                // 验证系统环境是否可以运行 Calico 节点实例
16
​
17
Options:
18
  -h --help      Show this screen.
19
​
20
Description:
21
  Node specific commands for calicoctl.  These commands must be run directly on
22
  the compute host running the Calico node instance.
23
​
24
  See 'calicoctl node <command> --help' to read about a specific subcommand.
Copied!
获取 Calico 节点状态信息：
1
$ sudo calicoctl node status
2
Calico process is running.
3
​
4
IPv4 BGP status
5
+--------------+-------------------+-------+----------+-------------+
6
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
7
+--------------+-------------------+-------+----------+-------------+
8
| 172.17.8.102 | node-to-node mesh | up    | 23:30:04 | Established |
9
+--------------+-------------------+-------+----------+-------------+
10
​
11
IPv6 BGP status
12
No IPv6 peers found.
Copied!
calicoctl node run calico 节点启动参数选项：
1
Usage:
2
  calicoctl node run [--ip=<IP>] [--ip6=<IP6>] [--as=<AS_NUM>]
3
                     [--name=<NAME>]
4
                     [--ip-autodetection-method=<IP_AUTODETECTION_METHOD>]
5
                     [--ip6-autodetection-method=<IP6_AUTODETECTION_METHOD>]
6
                     [--log-dir=<LOG_DIR>]
7
                     [--node-image=<DOCKER_IMAGE_NAME>]
8
                     [--backend=(bird|gobgp|none)]
9
                     [--config=<CONFIG>]
10
                     [--no-default-ippools]
11
                     [--dryrun]
12
                     [--init-system]
13
                     [--disable-docker-networking]
14
                     [--docker-networking-ifprefix=<IFPREFIX>]
15
                     [--use-docker-networking-container-labels]
16
​
17
Options:
18
  -h --help                Show this screen.
19
     --name=<NAME>         The name of the Calico node.  If this is not
20
                           supplied it defaults to the host name.
21
                           // 指定 Calico 节点名，如果没有指定则默认主机名
22
     --as=<AS_NUM>         Set the AS number for this node.  If omitted, it
23
                           will use the value configured on the node resource.
24
                           If there is no configured value and --as option is
25
                           omitted, the node will inherit the global AS number
26
                           (see 'calicoctl config' for details).
27
                           // 设置当前节点的 AS number，如果未指定，默认使用全局 As number
28
     --ip=<IP>             Set the local IPv4 routing address for this node.
29
                           If omitted, it will use the value configured on the
30
                           node resource.  If there is no configured value
31
                           and the --ip option is omitted, the node will
32
                           attempt to autodetect an IP address to use.  Use a
33
                           value of 'autodetect' to always force autodetection
34
                           of the IP each time the node starts.
35
                           // 设置当前节点本地 IPv4 路由地址，如果未指定，
36
                           // 则使用节点资源配置的值，如果也未配置，则自动探测使用地址
37
     --ip6=<IP6>           Set the local IPv6 routing address for this node.
38
                           If omitted, it will use the value configured on the
39
                           node resource.  If there is no configured value
40
                           and the --ip6 option is omitted, the node will not
41
                           route IPv6.
42
                           // 设置当前节点本地 IPv6 路由地址，如果未指定，
43
                           // 则使用节点资源配置的值，如果也未配置，则不会路由 IPv6
44
    ... ...
45
     --log-dir=<LOG_DIR>   The directory containing Calico logs.
46
                           [default: /var/log/calico]
47
                           // 指定 Calico 日志存储目录，默认为 /var/log/calico
48
     --node-image=<DOCKER_IMAGE_NAME>
49
                           Docker image to use for Calico's per-node container.
50
                           [default: calico/node:%s]
51
                           // 指定节点镜像
52
     --backend=(bird|gobgp|none)
53
                           Specify which networking backend to use.  When set
54
                           to "none", Calico node runs in policy only mode.
55
                           The option to run with gobgp is currently
56
                           experimental.
57
                           [default: bird]
58
                           // 指定网络存储类型，gobgp 当前处于实验性阶段，默认使用 bird
59
     --dryrun              Output the appropriate command, without starting the
60
                           container.
61
                           // 只输出执行命令信息，而不启动容器
62
     --init-system         Run the appropriate command to use with an init
63
                           system.
64
                           // 使用 init system 运行命令
65
     --no-default-ippools  Do not create default pools upon startup.
66
                           Default IP pools will be created if this is not set
67
                           and there are no pre-existing Calico IP pools.
68
                           // 启动不创建默认的 IP 池
69
     --disable-docker-networking
70
                           Disable Docker networking.
71
                           // 停用容器网络
72
     --docker-networking-ifprefix=<IFPREFIX>
73
                           Interface prefix to use for the network interface
74
                           within the Docker containers that have been networked
75
                           by the Calico driver.
76
                           [default: cali]
77
                           // docker 容器接口前缀，默认 cali
78
    ... ...
79
  -c --config=<CONFIG>     Path to the file containing connection
80
                           configuration in YAML or JSON format.
81
                           [default: /etc/calico/calicoctl.cfg]
82
                           // 配置文件路径，默认 /etc/calico/calicoctl.cfg
Copied!
注：经测试，此处 --name 选项必须为主机名，否则和 bgppeer 的 node 字段匹配不上，bgppeer 的 node 字段必须为主机名，后续还需进一步测试。
资源类型
资源结构概览：
1
apiVersion: v1                      // API 版本号
2
kind: <type of resource>            // 资源类型
3
metadata:                           // 元数据
4
  # Identifying information
5
  name: <name of resource>
6
  ...
7
spec:                               // 资源配置信息
8
  # Specification of the resource
9
  ...
Copied!
bgpPeer
​Calico’s documentation on L3 Topologies​
配置 Calico 集群节点，支持如下别名：bgppeer、bgppeers、bgpp、bgpps、bp、bps
1
apiVersion: v1
2
kind: bgpPeer
3
metadata:
4
  scope: node           // 范围：global/node
5
  node: rack1-host1     // 节点对应的主机名，如果是 scope 为 global，则此行必须省略
6
  peerIP: 192.168.1.1   // 当前 peer ip 地址
7
spec:
8
  asNumber: 63400       // 当前 peer As Number
Copied!
v3.0 apiVersion 已变更为 projectcalico.org/v3
Field
Description
Accepted Values
Schema
scope
Determines the Calico nodes to which this peer applies.
global, node
string
node
Must be specified if scope is node, and must be omitted when scope is global.
The hostname of the node to which this peer applies.
string
peerIP
The IP address of this peer.
Valid IPv4 or IPv6 address.
string
此处 node 官档标明为主机名，另外实际测试中如果此处指定 calico node 启动时指定的节点名「非主机名」时，跨容器路由会有问题，所以此处必须标注为主机名。
关于 BGP 的一些术语：
在 BGP 网络中，所有参与 BGP 进程的路由器都称为 BGP-speaking 路由器（BGP-speaking 可以看成是 BGP 会话的意思）
对于活动的 BGP-speaking设备，称为 peer设备，它与其他 BGP-speaking 设备之间有一个活动的 TCP 连接。BGP speaker是指本地 BGP 路由器，而 peer（对等，或者对端）是指任何其他 BGP-speaking 网络设备
当 BGP peer 路由器位于不同 AS 中时，它们之间互称对方为外部 peer，当它们位于同一个 AS 中时，则称为内部 peer
当在 peer 设备间（也就是相互直接连接的 BGP 路由器之间）建立了 TCP 连接，每个 BGP peer 就会立即与对端交换所有的路由表，也就是完整的 BGP 路由表
Host Endpoint Resource (hostEndpoint)
Host Endpoint 资源表示运行 Calico 主机关联接口，每个主机的 endpoint 包括针对该接口设置 labels 和 profiles，用以应用相关策略。
1
apiVersion: v1
2
kind: hostEndpoint
3
metadata:
4
  name: eth0
5
  node: myhost
6
  labels:
7
    type: production
8
spec:
9
  interfaceName: eth0
10
  expectedIPs:
11
  - 192.168.0.1
12
  - 192.168.0.2
13
  profiles:
14
  - profile1
15
  - profile2
Copied!
IP Pool Resource(ipPool)
定义 Calico IP 地址资源池，除了 ipPool 还有以下别名：ippool、ippools、ipp、ipps、pool、pools
1
apiVersion: v1
2
kind: ipPool
3
metadata:
4
  cidr: 10.1.0.0/16
5
spec:
6
  ipip:
7
    enabled: false
8
  nat-outgoing: true
9
  disabled: false       # 标注为 true 表示不启用该地址池
Copied!
ipip：ipip tunneling configuration for this pool. If not specified, ipip tunneling is disabled for this pool. 在公有云平台跨主机通信需要添加这一选项
nat-outgoing：When enabled, packets sent from calico networked containers in this pool to destinations outside of this pool will be masqueraded。简单说，使得容器可以访问外网
如果直接和内网交换机打通，则去除 nat-outgoing 选项，否则容器访问外部网络还是以 nat 方式出去的。
如果 ip 池启用了 ipip，建议同时也开启 nat-outgoing。否则当工作负载和运行 Calico 的主机之间没有 nat-outgoing 路由时启用 ipip 是不对称的，并且可能导致流量由于 RPF 检查失败而被过滤。
Node Resource (node)
定义节点资源，除了 node，还可以使用 nodes、no、nos
默认启动 Calico node 实例，会自动创建一个使用主机名的节点资源。
1
apiVersion: v1
2
kind: node
3
metadata:
4
  name: node-hostname
5
spec:
6
  bgp:
7
    asNumber: 64512
8
    ipv4Address: 10.244.0.1
9
    ipv6Address: 2001:db8:85a3::8a2e:370:7334
Copied!
获取节点信息：
1
# calicoctl get node  -o wide
2
NAME                   ASN     IPV4            IPV6
3
host1                  64511   192.168.1.1
4
... ...
Copied!
Policy Resource(policy) 和 Profile Resource(profile)
关于规则的设置，当前还没有实际使用，具体可参考官网内容 Policy Resource (policy) 和 Profile Resource(profile)。