Different Spatial Scales of Shape Similarity Representation in Lateral and Ventral LOC. Drucker, Aguirre. Cerebral Cortex 2009

Uses the same stimulus as earlier study of Fourier basis and shape

1. ” The results, indicating a coarse spatial coding of shape features in lateral LOC and a more focused coding of the entire shape space within ventral LOC, may be related to hierarchical
   models of object processing.”
2. Small regions of cortex respond to different classes of images, and furthermore “… small regions of cortex contain populations capable of representing the entire space of images in a category”
3. “A counterpoint to this apparent specialization has been the demonstration that information regarding object category is also contained in the distributed pattern of voxel responses across
   and between these specialized regions (…).”
4. The results show there are also coarse-scale representations.  “This type of representation might
   correspond to the ‘‘chorus of fragments’’ model of Edelman and Intrator (1997), where individual properties of objects are represented by separate neural populations.”
5. “Our focus here is upon the representation of variations in stimulus identity within a simplified object category… In practice, the structure of a parameterized space of shapes can be recovered from human behavioral responses (e.g., reaction times or similarity judgments) …”
   1. This similarity may also be reflected in neural patterns, which is what they check out here
6. “Does a similar system of neural representation exist within human visual cortex? The human lateral occipital complex (LOC) shows similar functional properties to those previously ascribed to IT structures in the macaque. This region responds more strongly when a viewer is presented with images of parseable objects, as opposed to images that have no 2- or 3- dimensional interpretation, and appears largely indifferent to the method of object perception, for example, objects may be defined by luminance, texture, motion, or stereo difference”
   1. For IT and macaque, see this
   2. Ventral LOC is also called posterior fusiform sulcus (pFS)
7. “Two recent studies have demonstrated a relationship between the perceptual similarity and the
   distributed pattern of neural activity in LOC (Op de Beeck et al. 2008; Haushofer et al. 2008),”
8. During fMRI scanning, subjects viewed 16 different shapes defined by radial frequency components (RFCs; a series of sine waves of various frequencies describing perturbations from
   a circle; Zahn and Roskies 1972; Fig. 1)
   1. Idea of RFCs actually being used in shape recognition was eventually “experimentally rejected” but it makes convenient stimuli, also totally abstracted with no categorical boundaries
9. Shapes were modified by altering amplitude and phase of a particular frequency component
10. Here neural adaptation (depends on habituation) to shape is studied on neural level
11. “We asked in this study if the degree of recovery from neural habituation at different cortical sites was proportional to the transition in similarity between 2 stimuli.”
12. “In this study, we investigated if the distributed pattern of response can inform as to the identity
    of stimulus variation within an object category;”
13. “Continuous Neural Adaptation in Ventral LOC Is Proportional to Shape Similarity”
    1. No adaptation effects found in lateral LOC
    2. Magnitude in change in shape matched linearly with change in ventral LOC
14. “An alternative explanation for the proportional recovery from adaptation in ventral LOC is that the extreme stimuli (those from the corners of the stimulus space) may evoke a larger neural response generally (e.g., Kayaert et al. 2005). As the larger distance stimulus transitions tend to include these
    extreme stimuli to a greater extent, perhaps the apparent recovery from adaptation is actually a larger response to these extreme stimuli independent of an adaptation effect.”
    1. This is not the case, however, as the results show that “the proportional recovery from adaptation seen in ventral LOC indicates the presence of a population code for stimulus shape and cannot be attributed to a generally greater neural response to extreme stimuli.”
15. “Distributed Pattern Responses Distinguish between Shapes”
16. Use SVMs to analyze data at coarse spatial level, which worked well
17. “The accuracy of the SVM analysis and the identified patch within lateral LOC indicates that the distributed voxel pattern of activity in that area carries information about shape.However, the pattern difference between shapes need not reflect the similarity of the stimuli or indeed have any particular structure. The SVM requires only that patterns be different in order to distinguish them—no assumptions about similarity structure are made or used.”
18. “Within lateral LOC, the strongly discriminant responses seen in the SVM analysis were found to also reflect stimulus similarity consistently across subjects (t4 = 10.0, P = 0.001). In contrast,
    the distributed pattern of response in ventral LOC had a weaker correlation with the perceptual similarity of the stimuli (t4 = 1.2, P = 0.3) (Fig. 4A). The difference between these subregions of
    area LOC was significant (t4 = 11.4, P = 0.0003).”
    1. Mixed evidence for this being attributable strictly to retinotopic similarity
19. “The RFC-Amplitude and RFC-Phase Axes Are Differentially Represented at Coarse and Fine Neural Scales”
20. “Although the distributed neural similarity matrix measured from lateral LOC was strongly correlated with the stimulus similarity matrix, there appeared to be aspects of the structure
    of the neural response not evident in the stimulus matrix”
21. Earlier studies on these shapes showed results that had phase and amplitude being recognized as orthogonal and equally important but that wasn’t completely replicated here.  Results here say the dimensions are “equally perceptually salient”, but that they are not perceived equivalently
22. “…both aspects of the stimulus space [amplitude, phase] are represented by the within-voxel population code within ventral LOC… A rather different result was observed for the distributed
    pattern of response within lateral LOC. There, the distributed pattern across subjects reflected the shapes primarily in terms of RFC-amplitude but not RFC-phase”
23. “For example, clusters of neurons might represent the tightness of the ‘‘knobs’’ of the shapes (defined by RFC-amplitude) independent of the direction that those knobs point within the overall shape (defined by RFC-phase). RFC-amplitude and RFC-phase may be taken as similar to ‘‘feature’’ and ‘‘envelope’’ parameters of Op  de Beeck et al. (2008), respectively; we thus contribute a similar finding in that features are represented in the distributed pattern in lateral LOC much more reliably than the overall shape envelope.”
24. “Based upon the differential sensitivity to shape identity for the adaptation and distributed pattern methods, we argue that although both the lateral and ventral components of area LOC contain neural population codes for shape, the spatial scale of these representations differ. Specifically, the
    absence of a distributed pattern effect within ventral LOC is evidence for a homogeneous representation of the shape space, such that the average response of any one voxel does not differentiate between the shapes, whereas the presence of a distributed code and the absence of an
    adaptation effect in lateral LOC suggests that there is a heterogenous distribution of shape representation,”
25. “Within ventral LOC, no meaningful tuning for the shape space can be identified: The amplitude of the response is no different for different shapes. This indicates that ventral LOC voxels are broadly tuned for shape identity. In contrast, lateral LOC voxels show relatively narrow tuning: there is a progressive decline in the response of a voxel for shapes more distant from the shape for which the voxel is best tuned (which was frequently a stimulus from the edges of the stimulus space). Moreover, lateral LOC voxels appear more narrowly tuned for the RFC-amplitude, as compared with the RFC-phase dimension of the shape space, consistent with our previous observation”
26. “The narrow tuning observed in lateral LOC may also explain the absence of a linear adaptation response in this region to transitions in shape space. If a given voxel is narrowly tuned to a particular region of the shape space, then it may only show recovery from adaptation for stimulus transitions within its tuned area.”
27. <Discussion>
28. ” By using a continuous carryover design, our study was capable of examining neural similarity both on a coarse, across-voxel scale by distributed pattern analysis, as well as on a fine, within-voxel scale using continuous neural adaptation. We can thus compare the information provided at distributed and focal levels.”
29. “Unlike ventral LOC, the lateral portion of LOC did not show adaptation responses that were linearly related to shape similarity. We found that the narrow tuning of lateral LOC voxels could explain this finding, indicating that each particular voxel has a population of neurons that are tuned to one specific region of the shape space. Consequently, most of the transitions between stimuli would not induce neural adaptation within the voxel as they would be transitions between stimuli not within the voxel’s receptive field.”

Categorical Clustering of the Neural Representation of Color. Brouwer, Heeger. JNeuro 2013.

1. fMRI study where subjects viewed 12 colors did either a color-naming or distractor task
2. “A forward model was used to extract lower dimensional neural color spaces from the high-dimensional fMRI responses.”
3. Vision areas of V4 and V01 showed clustering for color naming task but not for distractor
4. “Response amplitudes and signal-to-noise ratios were higher in most visual cortical areas for color naming compared to diverted attention. But only in V4v and VO1 did the cortical representation
   of color change to a categorical color space”
5. We can perceive thousands of colors but have only a handful of descriptive categories for colors, so we can see two different colors but would still potentially call it the same thing
6. Inferotemporal cortex (IT) is believed to deal with categorization of color
7. “…performing a categorization task alters the responses of individual color-selective neurons in macaque IT (…).”
8. Similar colors cause overlapping patterns of neural activity, “… neural representations of color can be characterized by low-dimensional ‘neural color spaces’…”
9. “Activity in visual cortex depends on task demands (…).”
   1. Use fMRI to study this
10. “Forward model” is used to reduce fMRI signals to a lower dimensional space of bases
11. ” Normal color vision was verified by use of the Ishihara plates (Ishihara, 1917) and a computerized version of the Farnsworth–Munsell 100 hue scoring test (Farnsworth, 1957).”
    1. <Need to learn about this>
12. “The 12 stimulus colors were defined in DKL (Derrington, Krauskopf and Lennie) color space… We chose the DKL space because it represents a logical starting point to investigate the neural representation of color in visual cortex. Although there is evidence for additional higher-order color mechanisms in visual cortex (Krauskopf et al., 1986), the color tuning of neurons in V1 can be approximated by linear weighted sums of the two chromatic axes of DKL color space
    (Lennie et al., 1990).”
13. Color categorization task was done outside the scanner, involved putting 64 colors into one of 5 categories
14. When in the fMRI, there were two types of episodes.  In one, subjects had to press one of 5 buttons to categorize the color (R,G,B,Y, or purple).  Distractor task was a 2-back test (is color the same as the color 2 steps ago)
15. <details on fMRI processing>
16. Used the forward model from this paper.
17. ” We characterized the color selectivity of each neuron as a weighted sum of six hypothetical
    channels, each with an idealized color tuning curve (or basis function) such that the transformation from stimulus color to channel outputs was one to one and invertible. Each basis function was a half-wave-rectified and squared sinusoid in DKL color space.”
18. Assume voxel response is proportional to the number of responding neurons in that voxel
19. Channel responses C (an n x c matrix where n was number of colors, and c # channels(6)).  Then did PCA on this to “extract neural color spaces from the high-dimensional space of voxel responses(…)”
20. “According to the model, each color produces a unique pattern of responses in the channels, represented by a point in the six-dimensional channel space.  By fitting voxel responses to the forward model, we projected the voxel responses into this six dimensional subspace.”
    1. PCA <A competing method they previously used to do this analysis> did not work as well – had similar results but more variability because it tries to fit noise where the forward model throws it out
21. To visualize the forward model, they ran PCA to project the 6D space to 2D (these 2 dimensions accounted for almost all the variance)
22. “Reanalysis of the current data using PCA to reduce dimensionality directly from the number of voxels to two also yielded two-dimensional neural color spaces that were similar to those published previously. Specifically, the neural color spaces from areas V4v and VO1 were close to circular, whereas the neural color spaces of the remaining areas (including V1) were not circular, replicating our previously published results and supporting the previously published conclusions (Brouwer and Heeger, 2009).”
23. Used many different clustering methods to see if colors labeled in the same color category had a more similar response than those in other categories
24. On to results
25. Subjects were pretty consistent where they put color class boundaries.  Blue and green were the most stable
26. Subjects weren’t told category labels–basically that they were doing clustering–but still categories were intuitively identifiable and pretty stable
27. Color clustering was strongest in V01 and V4v, during the color-naming task.  Responses from neighboring area V3 were more smoothly circular and therefore not as good at clustering.
28. 
29. “The categorical clustering indices were significantly larger for color naming than diverted attention in all but one (V2) visual area (p 0.001, nonparametric randomization test), but the
    difference between color naming and diverted attention was significantly greater in VO1 relative to the other visual areas (p 0.01, nonparametric randomization test). One possibility is that all visual areas exhibited clustering of within-category colors, but that the categorical clustering indices were low in visual areas with fewer color-selective neurons, i.e., due to a lack of statistical power”
30. “… no visual area exhibited categorical clustering significantly greater than baseline for the diverted attention task.”
31. Manual clustering done by subjects matched that done from the neural data, aside from the fact that neurall turqoise/cyan matched with blues, whereas people matched it with greens
32. “Hierarchical clustering in areas V4v and VO1 resembled the perceptual hierarchy of color categories”
    1. In V01 when doing color naming.
    2. The dendogram resulting from the distractor task looks pretty much like garbage
33. <Shame on the editor.  Use SNR without defining the abbreviation – I assume its signal to noise ratio?>
34. “Decoding accuracies from the current data set were similar; forward-model decoding
    and maximum-likelihood decoding and were nearly indistinguishable.”
35. <Between this and the similarity of the result of PCA, what does their forward model buy you?  Is it good because it matches results and is *less* general?>
36. “… we propose that some visual areas (e.g., V4v and VO1) implement an additional color-specific change in gain, such that the gain of each neuron changes as a function of its selectivity relative to the centers of the color categories (Fig. 8C). Specifically, neurons tuned to a color near the center of a color category are subjected to larger gain increases than neurons tuned to intermediate colors”
    1. <It is only shown that doing this in simulation helps clustering, which is in the neural data, but they don’t show that the neural data specifically supports this over other approaches>
37. “Task-dependent modulations of activity are readily observed throughout visual cortex, associated with spatial attention, feature-based attention, perceptual decision making, and task structure (Kastner and Ungerleider, 2000; Treue, 2001; Corbetta and Shulman, 2002; Reynolds and Chelazzi, 2004; Jack et al., 2006; Maunsell and Treue, 2006; Reynolds and Heeger, 2009). These task-dependent modulations have been characterized as shifting baseline responses, amplifying gain and increasing SNR of stimulus-evoked responses, and/or narrowing tuning widths. The focus in the current study, however, was to characterize task-dependent changes in distributed neural representations, i.e., the joint encoding of a stimulus by activity in populations of neurons.”
38. <Need to read all references in section “Categorical specificity of areas V4v and VO1”>
39. Lots of results that show V4 and nearby areas respond to chromatic stimuli.  They have a previous paper (their one from 2009) that V4v and V01 better match perceptual experience of color than other regions, but there aren’t many results dealing with “… the neural representation of color categories, the representation of the unique hues, or the effect of task demands on these representations”
40. Previous EEG studies show that the differences in EEG when looking at one color and then another “…  appear to be lateralized, providing support for the influence of language on color  categorization, the principle of linguistic relativity, or Whorfianism (Hill and Mannheim, 1992; Liu et al., 2009; Mo et al., 2011). Indeed, language-specific terminology influences preattentive color perception. The existence in Greek of two additional color terms, distinguishing light and dark blue, leads to faster perceptual discrimination of these colors and an increased visual mismatch negativity of the visually evoked potential in native speakers of Greek, compared to native speakers of English (Thierry et al., 2009).”
    1. Here however, no evidence of lateralized categorical clustering from fMRI
41. Neural research on Macaques and color, but there are differences in brain structure and sensitivities in photoreceptors between them and us so we need to keep that in mind when examining the results from animal experiments on color
42. “We proposed a model that explains the clustering of the neural color spaces from V4v and VO1, as well as the changes in response amplitudes (gain) and SNR observed in all visual areas. In this model, the categorical clustering observed in V4v and VO1 is attributed to a color-specific gain change, such that the gain of each neuron changes as a function of its selectivity relative to the centers of the color categories.”

Decoding and Reconstructing Color from Responses in Human Visual Cortex. Brouwer, Heeger. JNeuro 2009

1. Tried to decode color from FMRI with “conventional pattern classification, a forward model of idealized color tuning, an d ” PCA
   1. The conventional classifier was able to match training data to colors, but the forward model was able to extrapolate to new colors
2. Color was decoded accurately from:
   1. V1, V2, V3, V4, and V01
   2. But not L01, L02, V3A/B or MT+
3. In V4 and V01, 1st 2 principcal components “revealed progression through perceptual color space” (closeness defined a special way)
   1. This similarity didn’t manifest itself anywhere else, even though classification may have been accurate, and classification was actually most accurate in V1, where this similarity effect didn’t manifest itself.
   2. “This dissociation implies a transformation from the color representation in V1 to reflect color space in V4 and V01.”
4. There is color sensitivity throughout visual cortex
5. Classification of visual information through fMRI has been done previously on object categories, hand gestures, and visual features
6. <mostly skipping notes on materials and methods>
7. Stimulus was a slowly drifting series of concentric rings<, actually a little unclear about this, the description of the colors, and motion are not clear to me>
8. Classification was done through an 8-way (8 colors originally presented) classifier, not some means of regression
9. “The first two principal components of the simulated cone-opponency responses revealed results similar to those observed in V1.”
10. “The forward model assumed that each voxel contained a large number of color-selective neurons, each tuned to a different hue.” <There are more details>
11. The cone-opponency model, however was worse at recreating a space that pushed all the colors apart, their forward model was successful at that, however
12. Forward model not only allowed for decoding, but also reconstructing stimulus colors from test data
13. <Skipping to discussion, running out of time>
14. Mean voxel responses themselves did not reliably distinguish color
15. Here saturation didnt vary, only hue varied
16. “Obviously, the lack of progression in the early visual areas (in particular V1) should not be taken as an indication that these areas are colorblind…  An alternative model of color selectivity, based on cone-opponent tuning rather than hue tuning, reproduced many features of the non-circular and self-intersecting color space derived from teh V1 PCA scores”
17. “…spatially distributed representations of color in V4 supported ‘interpolation’ to decode a stimulus color based on the responses to perceptually similar colors.”
18. “Nonetheless, our results support the hypothesis that V4 and VO1 play a special role in color vision and the perception of
    unique hues…”