OSPF MD5 authentication appears to fail with an unknown auth type 1026 when you specify an "instance ID" in BIRD, regardless of whether you specify the instance ID on the interface or not. Removing all occurrences of "instance ID" from bird.conf and re-committing the configuration appears to restore the correct auth type "cryptographic (2)". BIRD configuration and tcpdump info below.
protocol ospf v2 IPv4_OSPF { instance id 4; ecmp yes limit 8; area 0 { interface "eth3" instance 4 { type pointopoint; authentication cryptographic; password "<text removed>" { algorithm keyed md5; }; }; }; } Jul 25 17:31:31.124 UTC: ospf[1007]: Rcv pkt from 192.168.1.1, GigabitEthernet0/0/0/14: Mismatch Authentication type. Input packet specified type 1026, we use type 2 Frame 16: 94 bytes on wire (752 bits), 94 bytes captured (752 bits) Ethernet II, Src: [removed], Dst: IPv4mcast_05 (01:00:5e:00:00:05) Internet Protocol Version 4, Src: 192.168.1.1, Dst: 224.0.0.5 Open Shortest Path First OSPF Header Version: 2 Message Type: Hello Packet (1) Packet Length: 44 Source OSPF Router: 192.168.1.1 Area ID: 0.0.0.0 Checksum: 0x0000 (None) Auth Type: Unknown (1026) Auth Unknown: 000001105d39f2a3 OSPF Hello Packet