Without other usb devices the keyboard works immediately. With them it takes 5 ish seconds for it to start responding. This is regardless of which device is discovered first (which I can manipulate), it seems to just wait for all of them. Is there a way to make it not do that?
Hint: try another USB port.
USB is like a network, and it could be a jungle of cables and ports and switches and bridges and whatnot. But the closer you get to the mainboard, the better are your chances to escape such a problem.
This worked. Apparently I had all the usb devices connected to the same controller and it seems linux initialises them controller by controller. Thanks