HP Lovecraft liked to write of horrors from the beyond, describing them with terms such as "non-euclidian geometry" or "colors outside the visible spectrum". While these descriptions were meant to invoke a sense of terror and the unknowable, we actually come across colors outside the "visible spectrum" in our everyday lives and most people don't even know it.
The colors we can see in the electromagnetic spectrum range from a deep red up to a deep violet, with the ends fading off into infrared and ultraviolet respectively. While some animals can see a bit further into those that we can, we'll consider those truly impossible to see for now. These are not the impossible colors I'm hunting for today.
To detect and differentiate these ranges of frequencies, your eyes contain three separate type of cone cells, each type tuned to be more responsive to different frequencies in that spectrum than the other ones. To make things simple we consider them to be red, green, and blue, though they vary a little from person to person and each one responds to a range around each of those colors, and they overlap.
This graph shows the overlaps and how they're spaced.
Your brain and optics use the combinations of how these are triggered to try to figure out what color you are seeing. If red is strongly triggered and green less so, you see red. Red and green equally triggered? You're seeing yellow. Green and blue equally triggered? You see cyan.
Now, this leads to the first little trick we play with our eyes, and we use it every day. If yellow light comes into our eyes, it triggers red and green cone cells, and our brain sees it as yellow. If red and green light really close together comes into our eye and triggers the same red and green cone cells the same way... you also see yellow. This is why there's red, green, and blue pixels in computer monitors, it uses the way our eyes detect colors to generate those colors for us to see. Your monitor doesn't make violet or yellow or orange light, it makes ratios of red, green, and blue that makes our eyes think they see those colors.
So, back to the spectrum. Red and green triggered equally makes us see the frequency in between, yellow. Green and blue equally makes us see the frequency in between, cyan. Red and blue triggered equally... behaves completely different! Why?! Because, on the spectrum the frequency that would be between red and blue is green. But we already have a cone for green, and it's NOT triggering, so our brain knows something is wrong here. It needs to show us a color, but green isn't the right choice, so what does our brain do?
It MAKES SHIT UP! It says screw you, I'm gonna make a color that doesn't exist in nature, and I now present to you... MAGENTA! There is no frequency in the electromagnetic spectrum for magenta. It's not in a rainbow, it doesn't come out of a prism, you can't make a magenta colored laser by giving off specifically tuned photons, cause there's nowhere to tune to. The brain creates it from nothing to fill in a gap where the bottom and top of the spectrum would interact with each other.
So there's your first color that doesn't really exist, and you see it all the time. Well hopefully not ALL the time, it is a bit garish. But it's not unusual either.
The next set of impossible colors live where colors used to be. You're familiar with where they exist, we refer to them as afterimages. If you stare at one point for long enough, everything looks like it starts to fade away into non-color, like a wash of grey. What's happening is neuron fatigue, as the signals coming in stay constant the neurons being triggered by the cone cells in your eyes get used to the signal and start reacting less and less. So, you stop seeing them. Then, you looks somewhere else. It takes a while for that fatigue to recover, so your brain isn't really receiving the right signals from the red, green, and blue cones depending on which were fatigued. So, you see colors where there aren't really colors. In the right circumstances, you can see colors you couldn't based on just the light coming into your eyes.
An example of this is a color called stygian blue. Stare at a circle of bright yellow until your eyes start to fade. The green and red cones get tired and aren't responding. Now, look at something deeply black. Black has no light to trigger the cones, so you should see... black. But, your red and green neurons were already saying there was nothing there cause they stopped firing from fatigue. Now, they have to say there's even less there. So, even though there's no light coming in, there's more blue signal than there is anything else, so you see a deep, unworldly blue in the black, a color from nowhere. You can other stygian colors in the same way, using different original circle colors. Other impossible color families are luminous (when you see the afterimage on top of white, making it brighter than neon colors) and hyperbolic (when you layer the afterimage color on top of the color you're going to see, like staring at yellow and then looking at bright blue with the blue afterimage). Someone has put together a set of gifs and more info
at this Imgur post. Some people can see some types of these colors better than others. For instance, I can see luminous and stygian very easily, but it takes a long time for me to get any reaction trying the hyperbolic ones.
The final set of impossible colors require tricks to get around the very way the neurons that process the cone cell signals work. There are neurons that take the processed colors coming from your eyes, and say "is this more red or green?" and "Is this more yellow or blue?". The results of this give your brain a better idea if the hue you're looking at. Is it a yellowish green or a blueish green? But because of this, there are no such colors as a reddish green, or a bluish yellow. Or is there? You see, these neurons work with the cones in each eye. So, if you play a trick where one eye is seeing one of the colors and the other eye is seeing the other, you can sometimes tick the brain into combining them and showing you one of these two impossible hues. The most common way to try this is by looking at two side by side boxes in the two opposing colors and crossing your eyes to make them overlap.
This page has images set up that you can try your luck at. I've only managed it once before with yellowish-blue, hopefully you have better luck than me!
So, maybe Lovecraft was onto something. Maybe his colors out of space were magenta and stygian blue.