I was asked to give a Kantian perspective on this. Alas, Kant has little to contribute on this subject. He was not much interested in color either in epistemology or in aesthetics, and for the same reason: he thought of it as basically mere sensation, what he called the matter of empirical intuition or perception, and that everything we know about it we know empirically, and as only contingently true; and what he was interested in, both in epistemology and aesthetics, was form, the structure of perception that in his view was contributed by the mind and therefore could be known a priori, and as universally and necessarily valid. Indeed, both in epistemology (the "Transcendental Aesthetic" of the Critique of Pure Reason) and in aesthetics ("The Analytic of the Beautiful" of the Critique of the Power of Judgment) he used the case of color precisely to illustrate what he meant by the merely empirical element in perception, a matter for psychologists or physiologists but not philosophers.
And indeed, we should not think of philosophers as arm-chair psychologists or physiologists, so I will not myself engage in speculation about the physiology of color perception. The only thing I can say as a philosopher is that we have no particular reason to believe that the phenomenological effect of a cause that is some sense linear (the spectrum) itself needs to be linear -- way too many factors between external stimulus and consciousness for us to assume that!