If you know that you are not a zombie, then phenomenal character is conceptual and inverted spectra (color qualia being inverted relative to your conceptualizations) are impossible.
Regarding the supposition that you know that you are not a zombie, I interpret this as meaning that you know that you now have states with phenomenal character or qualia. It is useful to compare this kind of knowledge to more ordinary cases of knowing that something is the case. Take for example, my knowing that there is a dog in the room. In order for me to know this, there must be some set of properties that the dog has and that I am able to conceptualize. I can be relatively neutral on exactly which conceptualizations will get the job done. Maybe my conceptualization is that there’s a four-legged furry barker in the room. Maybe my conceptualization is that there’s a domesticated wolf-descendant in the room. Maybe my conceptualization is simply that there’s a dog in the room. But however it goes, there must be some set of properties of the dog (e.g. being domesticated, being wolfish) and I must have some set of concepts adequate for the accurate representation of those properties (e.g. the concept of domestication, the concept of wolves).
Now, my knowledge that I now have states with phenomenal character is seldom if ever analogous to the case in which I simply conceive of the dog as a dog. I am not now simply conceiving of myself as having phenomenal states. There are specific phenomenal states that I conceive myself as having. As I type this note and take breaks to sip coffee there’s a whole slew of qualia that I conceive my states as having. In particular, I conceive myself as seeing my coffee mug as being blue. I have a blue quale and am able to conceptualize it as such. I reject, then, the statement that there is no absolute correct orientation of the color spectrum. I think there is. It involves conceptualizing a blue quale as blue and a yellow quale as yellow and so forth.
Now, if qualia are distinct from my conceptualizations, as they would need to be if inverted spectra are possible, then it would be theoretically possible for my qualia to become inverted without my noticing. My quale that I currently conceptualize as blue would actually be yellow and vice versa. My current conceptualization as having a blue quale would be false, then. And it would be false without my noticing. Further, if qualia are distinct from my conceptualizations, I could have all the same conceptualizations without having any qualia at all, and my belief that I’m not a zombie would be false. If it’s possible for my belief that I’m not a zombie to be false, then I can’t know that I’m not a zombie. Thus does self-knowledge of non-zombie-hood lead to the impossibility of inverted spectra.