Open Infrastructure

h1. How to configure mikrotik network equipment

{{toc}}

h2. Status

This document is **PRE PRODUCTION**.

h2. Finding a directly connected Mikrotik switch

If you don't know the IP address of a Mikrotik switch, just connect a direct cable to it and ping the IPv6 multicast all nodes address:

<pre>

5: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 80:1f:02:d6:4c:50 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::821f:2ff:fed6:4c50/64 scope link 

       valid_lft forever preferred_lft forever

bridge:~# ping ff02::1%eth1

PING ff02::1%eth1 (ff02::1%5): 56 data bytes

64 bytes from fe80::821f:2ff:fed6:4c50: seq=0 ttl=64 time=0.115 ms

64 bytes from fe80::c6ad:34ff:fe88:832b: seq=0 ttl=64 time=0.588 ms (DUP!)

64 bytes from fe80::821f:2ff:fed6:4c50: seq=1 ttl=64 time=0.109 ms

64 bytes from fe80::c6ad:34ff:fe88:832b: seq=1 ttl=64 time=0.432 ms (DUP!)

^C

--- ff02::1%eth1 ping statistics ---

2 packets transmitted, 2 packets received, 2 duplicates, 0% packet loss

round-trip min/avg/max = 0.109/0.311/0.588 ms

bridge:~# 

</pre>

* One of the ip address is you, the other one is the switch

Connecting to it via ssh:

<pre>

[15:21] bridge:~% ssh admin@fe80::c6ad:34ff:fe88:832b%eth1

admin@fe80::c6ad:34ff:fe88:832b%eth1's password: 

</pre>

</pre>

h2. Setting up a newly arrived Mikrotik switch

This part is specific for mikrotik-crs326 devices and should

After arriving within 1 work day do:

* Unpack

* Find out which name the switch should have

** go to https://netbox.ungleich.ch

** search for crs326

** identify the last used number

** Create a new device

*** go to devices 

*** devices

*** add

**** name: "mikrotik-crs326-XX"

**** device-role: a device role

**** device-type: CRS326

**** serial-number: <from the device>

**** site: placeX -

**** create

* Add a physical label with its name

** Do not continue before you have done that!

* configure your notebook with the ipv4 address 192.168.88.23/24 

* connect to the crs326: @ssh admin@192.168.88.1@

* Configure the switch

** Set identity: @/system identity set name=mikrotik-crs326-XX@ # use the correct name

** Set password: @/user set admin password=@ # use the password for mikrotik in the place that you are, use password store

** Enable IPv6: @/system package enable ipv6@

** Reboot so that IPv6 is enabled: @/system reboot@

** Add a place indendent, unrouted IPv6 address: @/ipv6 address add eui-64=yes advertise=no interface=bridge address=2a0a:e5c0:1:c::/64@

** Get the assigned IPv6 address: @/ipv6 address print@

** Disconnect from the switch

* Assign your notebook the IPv6 address *2a0a:e5c0:1:c::23/64*

* Connect to the switch via IPv6

** Remove the IPv4 address 192.168.88.1 to avoid collisions with other switches: @/ip address remove numbers=0@

*** Verify: @/ip address print@

* Verify / update entries in netbox:

** Verify that the serial number is correct: @:put [ /system routerboard get serial-number ]@

** Netbox

*** go to the ipv6 prefix https://netbox.ungleich.ch/ipam/prefixes/216/

**** create a new ipv6 address

**** set the DNS name to *mikrotik-crs326-XX*

**** role = loopback

* Ensure all steps are taken correctly

* Move the device to our inventory / stock if not directly used

h2. How to configure VLANs on the Mikrotik CRS326 series

* Do not use the standard Linux approach of putting vlan interfaces into bridges

* Switches can suddenly stop working

** Compare https://wiki.mikrotik.com/wiki/Manual:Layer2_misconfiguration

*** Specifically this: https://wiki.mikrotik.com/wiki/Manual:Layer2_misconfiguration#VLAN_in_bridge_with_a_physical_interface

Instead use the following procedure outlined in https://wiki.mikrotik.com/wiki/Manual:Basic_VLAN_switching#CRS3xx_series_switches

<pre>

/interface bridge add name=bridgevlans

# Tagged interfaces need to go in like this:

/interface bridge port add bridge=bridgevlans interface=sfp-sfpplus1 hw=yes

# Untagged interfaces need to go in like this:

/interface bridge port add bridge=bridgevlans interface=ether2 hw=yes pvid=20

/interface bridge port add bridge=bridgevlans interface=ether3 hw=yes pvid=20

...

# Then add them to "bridge vlan"

/interface bridge vlan add bridge=bridgevlans tagged=ether1 untagged=ether2,ether3 vlan-ids=20

/interface bridge vlan add bridge=bridgevlans tagged=ether1,bridge1 vlan-ids=99

# Management

/interface vlan add interface=bridgevlans vlan-id=99 name=MGMT

/ipv6 address add eui-64=yes advertise=no address=2a0a:e5c0:... interface=MGMT

# Last step

/interface bridge set bridge1 vlan-filtering=yes

# Might need reboot for activating the IPv6 address

/system reboot

</pre>

h2. Configuring a Mikrotik switch for integration into the network

Depending on the usage scenario, we will define a variety of bridges and assign ports to it.

* In almost all cases the uplink port will be the *sfp-sfpplus1* port.

* In almost all cases we want to apply vlan tagging to that port

* For server networks, we configure the MTU to the highest value the switch supports

** Devices are configured with a 9200 MTU

** So the MTU on the switch ports needs to be at least 9200

h3. First step for all integrations

Setup MTU of all *sfp-sfpplus* ports (plural) to 9200 and the L2MTU to 9204:

Use @/interface print@ to find out the maximum MTU:

<pre>

[admin@mikrotik-crs326-1] > /interface print 

Flags: D - dynamic, X - disabled, R - running, S - slave 

 #     NAME                                TYPE       ACTUAL-MTU L2MTU  MAX-L2MTU MAC-ADDRESS      

 0   S ether1                              ether            9200  9204      10218 B8:69:F4:8E:AC:BA

 1   S ether2                              ether            9200  9204      10218 B8:69:F4:8E:AC:BB

...

24  R  sfp-sfpplus1                        ether            9200  9204      10218 B8:69:F4:8E:AC:D2

25     sfp-sfpplus2                        ether            9200  9204      10218 B8:69:F4:8E:AC:D3

</pre>

Above is already setup, but the values differ, use the following to set it correctly:

<pre>

/interface set sfp-sfpplus1 mtu=9200 l2mtu=9204

/interface set sfp-sfpplus2 mtu=9200 l2mtu=9204

</pre>

h3. Use case 1: coworking network

To use the switch in a coworking network, we keep the MTU

* Create a vlan interface named *vlan-coworking*: @/interface vlan add  vlan-id=15 name=vlan-coworking interface=sfp-sfpplus1@

** Verify that it is created: @/interface vlan print@

** Lookup the vlan-id from https://netbox.ungleich.ch/ipam/vlans/

** The vlan will be added to *sfp-sfpplus1*

* Create a bridge named *bridge-coworking*: @/interface bridge add name=bridge-coworking@

** Verify that it is created: @/interface print@

* Add the interface *vlan-coworking* to the *bridge-coworking* as a port: @/interface vlan add interface=vlan-coworking@

** Verify that the port is added: @/interface vlan print@

* Add all ethernet ports that you need for coworking to the *bridge-coworking*

** For instance to add port 7: @/interface bridge port set bridge=bridge-coworking numbers=6@ 

*** Find the number of the interface with @/interface bridge port print@

** Verify again like you did above

h3. Use case 2: server or internal network

Steps similar to above, BUT ensure that the MTU is set correctly on all interfaces.

* Ensure that the mtu is correct on the *sfp-sfplus* interfaces (see above)

** This is important

* We create a bridge named

** *bridge-server*: @/interface bridge add name=bridge-server@

** *bridge-internal*: @/interface bridge add name=bridge-internal@

* We create a new vlan interface on *sfp-sfpplus1* named

** *vlan-server*: @/interface vlan add name=vlan-server interface=sfp-sfpplus1 mtu=9200@

** *vlan-internal*: @/interface vlan add name=vlan-internal interface=sfp-sfpplus1 mtu=9200@

* Add the

** *vlan-server* interface as a port to *bridge-server*: @/interface  bridge port add interface=vlan-server bridge=bridge-server@

** *vlan-internal* interface as a port to *bridge-internal*: @/interface  bridge port add interface=vlan-internal bridge=bridge-internal@

* Verify that the MTU is correct

** On the VLAN interface: @/interface vlan print@

** On the bridge: @/interface bridge print@

** On the sfp-sfpplus interfaces: @/interface print@

<pre>

If the MTUs are wrong, you can encounter hanging connections, while ping (small packet) still works. Be careful to do it right.

</pre>

h2. Typical setup for a new mikrotik-crs326 in place6

You need to insert a GBIC module *BEFORE* seting mtu on the sfp-sfpplus interface

<pre>

/interface set sfp-sfpplus1 mtu=10216 l2mtu=10218

/interface bridge add name=bridge-server

/interface vlan add name=vlan-server interface=sfp-sfpplus1 mtu=10214 vlan-id=11

/interface bridge port add interface=vlan-server bridge=bridge-server

/interface bridge add name=bridge-internal

/interface vlan add name=vlan-internal interface=sfp-sfpplus1 mtu=10214 vlan-id=10

/interface bridge port add interface=vlan-internal bridge=bridge-internal

/ipv6 address add eui-64=yes advertise=no interface=bridge-internal address=2a0a:e5c0:2::/64

/ipv6 address print

</pre>

h2. Update the switch to the latest version

(TBD for IPv6 only networks)

h2. Make switch accept router advertisements!

<pre>

[admin@mikrotik-crs236-2] > /ipv6 settings set accept-router-advertisements=yes

</pre>

FINALLY!