I have two main options:
Beware that since the Pi 2 is newer, most discussions of cross-compiling revolve around the A/B/+ (aka. "Pi 1") models. The difference is that the 2's BCM2836 processor is ARMv7 based, whereas the BCM2835 used on older models is ARMv6 based.
Since ARMv7 is backward compatible with ARMv6, you can use binaries intended for the A/B/+ on a Pi 2. However, if you want to fully exploit the architecture, you would want to target ARMv7 specifically.
ARMv6 is relatively obscure, but thanks to the explosion of smartphones and other mobile devices over the past half decade, ARMv7 is very widespread. Hence, finding an ARMv6 cross-compiler is difficult (which is why the options for targeting A/B/+ models are few), whereas finding an ARMv7 cross-compiler is very easy -- all mainstream GNU/Linux distros have one available. So you don't need to build your own or rely on Linaro, you can just install via apt/yum/pacman/whatever on on your x86 box and use that.
Is it something that can make any visible difference at all?
I won't claim to be an expert on the nitty gritty details, but since Linaro and pretty much anything else is bound to be gcc based, I doubt it will matter that much. If you find something that isn't gcc based from which you plan to scratch build a linux system, I hope you know what you are doing, and good luck. Clang might be an exception (but I don't think it is considered to produce a better result).
In any case, you are free to experiment and change your mind later. I would just start with a distro cross-compiler for now.