plugin-programmable-search-engine (a README Experience)

A plugin for Micro.blog for adding a site search interface using Google’s programmable search engine API. It’s code lives here

Creating your Google Programmable Search Engine

You will need a Google account. If you don't have one, go ahead and create one. Once you're logged in, follow these steps to configure yourself a search engine.

  1. Navigate to the landing page and click the Get Started button.

  2. Click the Add button to create a new instance.

  3. Enter your site address (I believe I went with *.moondeer.blog/*) and click CREATE.

  4. Click on Control Panel to get to the configuration.

  5. Here’s the first value we need, the Search engine ID, this is the value referred to every else as cx.

  6. Now, scroll down a bit and find the Programmatic Access section. Click the Get Started button to the right of Custom JSON Search API.

  7. Find and click the Get a Key button.

  8. Select the + Create a new project option.

  9. Pick a project name and click NEXT.

  10. And here’s value number two, the API key.

Test Driving your Engine

If you follow those ten steps, you’ll end up with a working search engine constrained to your site content (or what Google has crawled of it). You can navigate to that Public URL Google kept mentioning as you were getting things setup. If you do, you’ll find an unimpressive page looking something like this:

Go ahead and perform a search. These are the same results the plugin will have access to using that Custom Search JSON API. The image below explains why I took one look at the Google-supplied interface options and said, “F$&k that noise … how do we get a hold of this data to display for ourselves”:

Data, What Data?

So that Custom Search JSON API page we visited to get our API key kept pushing you to Try this API, am I right? Here is how you can. Locate that Search engine ID value that I warned you we’d start calling cx and plop that little f$&ker in the cx field kinda like:

Then scroll until you find the q field (because query is such a long f$&king word) and plop in some search terms kinds like:

Now, the first time I did this (right after creating my API key) I got an error. No idea why. If you get one, holler. If you don’t, it means you have access to the exact same data the plugin (and that Public URL page) pulls down.

The response to a custom search request is detailed here. Let’s go over the bits I found to be relevant.

The Relevant Response Bits

queries
queries contains 1 - 3 collections of metadata that amount to a glorified page index. There will always be an entry for request, this amounts to the current page. Each collection contains a startIndex value. I did a whole thing where I generated numbered navigation links for the pages at the bottom (you know, as you do); but, it turns out that the f$&king totalResults value is dynamic for some f$&king reason and displaying indices as estimates is horsesh$te (when I checked the Public URL page to see if Google, itself, could manufacture accurate indices, I found they could not. Run through your result pages, if you like swing-and-a-miss-indices, I will put them back). Anyway, all we really care about (seeing as we know the search terms), is that startIndex and the abscence or presence of previousPage and nextPage entries (and their startIndex values). If I’ve missed something you find to be relevant, feel free to share.
"queries": {
  "previousPage": [{
    …
    "startIndex": integer,
    …
  }],
  "request": [{
    …
    "startIndex": integer,
    …
  }],
  "nextPage": [{
    …
    "startIndex": integer,
    …
  }]
}
items
The items array holds the search result items corresponding to queries.request. The bits the plugin currently utilizes are these:
{
  …
  "title": string,
  …
  "link": string,
  "displayLink": string,
  …
  "htmlSnippet": string,
  …
  "pagemap": {
    f$&kin-random
  },
  …
}

It’s the top four properties that compose the bulk of the displayed item. For result items without an image, they compose it entirely.

None of my result items have ever had an image entry at the top level as depicted in the API (If y’all end up with an entry, holler, and I’ll add checks for the image property). The thumbnail images are pulled out of that pagemap entry. This collection of page metadata will be influenced by Micro.blog theme and Micro.blog plugin installations. The result items I have received consistently contain an entry at pagemap.cse_thumbnail[0] (yeah, they wrap everything in a f$&kin’ array, no idea why). The plugin looks for this entry and when it finds it you get a thumbnail image to go along with the other stuffs.

So What the F$&k is the PageMap?

Well, it turns out that a PageMap is yet another f$&kin’ form of structured data used by Google. It consists of a butt-ugly chunk of XML injected into the page <head> inside an HTML comment.

<html>
  <head>
   ...
  <!--
  <PageMap>
     <DataObject type="document">
        <Attribute name="title">The Biomechanics of a Badminton
        Smash</Attribute>
        <Attribute name="author">Avelino T. Lim</Attribute>
        <Attribute name="description">The smash is the most
        explosive and aggressive stroke in Badminton. Elite athletes can
        generate shuttlecock velocities of up to 370 km/h. To perform the
        stroke, one must understand the biomechanics involved, from the body
        positioning to the wrist flexion. </Attribute>
        <Attribute name="page_count">25</Attribute>
        <Attribute name="rating">4.5</Attribute>
        <Attribute name="last_update">05/05/2009</Attribute>
     </DataObject>
     <DataObject type="thumbnail">
        <Attribute name="src" value="http://www.example.com/papers/sic.png" />
        <Attribute name="width" value="627" />
        <Attribute name="height" value="167" />
     </DataObject>
  </PageMap>
  -->
  </head>
   ...
</html>

Can we utilize PageMaps to inject precisely that data which every Micro.blogger using this plugin would want to have availble for display when viewing their search results? Jury is still out. I did construct a partial that injects such an eye sore. For a post with all the fixings it generates something kinda like:

<!--
<PageMap>
  <DataObject type="post">
    <Attribute name="title">On the American Upside Down</Attribute>
    <Attribute name="summary">While the beltway press, the pundits, the 
      influencers, the organizers, and the cogs that compose the political
      machinery at large desperately cling to those norms and precedents 
      with which order has so long been coaxed from chaos, the unprecedented
      leaves breadcrumbs for the fresh-eyed to trace towards the trailhead,
      near the clearing within which it has planted itself openly as 
      invitation for observation.</Attribute>
    <Attribute name="reading_time">8</Attribute>
    <Attribute name="category">Perspectives</Attribute>
    <Attribute name="publish_date">2021-09-28T15:24:00-08:00</Attribute>
    <Attribute name="modified_date">2021-09-28T15:24:00-08:00</Attribute>
  </DataObject>
  <DataObject type="image">
    <Attribute name="src" value="https://moondeer.blog/uploads/2021/cbeaf7bb2f.jpg" />
  </DataObject>
  …
  <DataObject type="image">
    <Attribute name="src" value="https://moondeer.blog/uploads/2021/de854e7aa6.jpg" />
  </DataObject>
</PageMap>
-->

Did it work? Time will tell. I assume we have to wait for Google to re-crawl the site’s pages, at which point the entries will show up in pagemap or they won’t.

Configuration

As with all my plugins, this plugin is configured by the documented data templates under the data directory (Check out the other READMEs or holler at me for more on this). This particular plugin has four configuration files located under the plugin_programmable_search_engine subdirectory (here's a hint, recreate these files in a custom theme, changing the subdirectory to plugin-programmable-search-engine and your data will persist between plugin updates). Let's look at each file and then call it a day.

Config.toml

This is the only file that the plugin requires you to edit in order to function properly. It looks like this:

# Debug and build related parameters
####################################

# Theme version, printed to HTML comment when the plugin loads.
#
Version = '2.0.3'

# Whether to print HTML comments for debugging purposes.
#
DebugPrint = false

# Whether to provide subresource integrity by generating a 
# base64-encoded cryptographic hash and attaching a .Data.Integrity
# property containing an integrity string, which is made up of the
# name of the hash function, one hyphen and the base64-encoded hash sum.
#
Fingerprint = true

# Output style for the generated stylesheet.
# Valid options are nested, expanded, compact and compressed
#
SassOutput = 'compact'

# Whether to minify the generated Javascript file.
#
MinifyScript = false

# Whether to inject a page map structure into the page <head>
# in the hopes that Google will eventually parse and provide it.
#
InjectPageMap = true

# Search engine configuration
#############################

# The search engine identifier
#
CX = ""

# The search engine API key
#
APIKey = ""

It's the CX and APIKey values that you must fill in. You can prevent the injection of that experimental page map XML by setting InjectPageMap to false. Version is utilized by an HTML comment that gets injected into the page <head>. DebugPrint allows you to dump the parameter values as parsed by the plugin into an HTML comment injected into the page <head>. The other three parameters control the script and stylesheet builds.

The remaining three files each control the style of one of the plugin's visual components: the search bar, the results container, and the individual result items. Let's briefly look at each of those and then I'll show y'all how the stylesheet gets generated.

SearchBar.toml

The file used to configure the search bar looks like this:

# Parameters controlling the search bar injection and style
###########################################################

# The ID of the element that will serve as the parent of the 
# search bar injected via Javascript
#
ContainerID = 'pse-container'

# The ID assigned to the search bar.
#
ID = 'pse-search-bar'

# Styling the input field
#########################

# The ID assigned to the input field.
#
Input.ID = 'pse-search-bar-input'

# Whether the input field of the search bar is collapsible.
#
Input.Collapsible = true

# Whether the input field is initially expanded or collapsed
#
Input.InitialState = 'collapsed'

# The height to set for the input field
#
Input.Height = 'auto'

# The width to set for the input field
#
Input.Width = '300px'

# The color of text within the input field
#
Input.Color = 'black'

# The placeholder text displayed when the input field is empty
#
Input.Placeholder.Text = 'site search'

# The color of the placeholder text
#
Input.Placeholder.Color = 'darkgray'

# The duration to use when collapsing and expanding the search field
#
Input.Transition.Duration = '.35s'

# The time function to use when collapsing and expanding the search field
#
Input.Transition.TimingFunction = 'ease'

# The vertical padding to set on the input field
#
Input.Padding.Y = '0'

# The horizontal padding to set on the input field
#
Input.Padding.X = '.5em'

# The border radius value to set on the input field
#
Input.Border.Radius = '1em'

# The border style to set on the input field
#
Input.Border.Style = 'solid'

# The border color to set on the input field
#
Input.Border.Color = 'rgba(black,.125)'

# The border width to set on the input field
#
Input.Border.Width = '1px'

# Styling the search bar button
###############################

# The ID assigned to the button.
#
Button.ID = 'pse-search-bar-button'

# The color to set for the button when the input is collapsed
#
Button.Color = 'lightgray'

# The horizontal padding to set on the button
#
Button.Padding.X = '.25em'

# The vertical padding to set on the button
#
Button.Padding.Y = '0'

While you can modify the ID values, these are here primarily to ensure consistency between the Javascript and the Sass / CSS. You will need to create an HTML element somewhere on your page and set its id value equal to ContainerID to act as the container for the search bar. You can control whether the search bar collapses and expands with Input.Collapsible, and you can set its initial state with Input.InitialState. The parameter comments oughta be enough to get your head wrapped around the remaining parameters.

ResultsOverlay.toml

This file is very much like the previous one accept that it is all style (no behavior modification). It looks like this:

# Styling the results overlay
#############################

# The ID assigned to the results overlay.
#
ID = 'pse-results-overlay'

# The ID assigned to the <article> element with the search results.
#
Article.ID = 'pse-results-article'

# The text color to set on the <article> element.
#
Article.Color = 'inherit'

# The font size to set on the <article> element.
#
Article.Font.Size = '1rem'

# The base background color
#
Article.BG.Color = 'white'

#Styling the <header> element of the results overlay
##################################################

# The ID assigned to the <header> element.
#
Header.ID = 'pse-results-header'

# The color for header text
#
Header.Color = 'currentcolor'

# The ID assigned to the title.
#
Header.Title.ID = 'pse-results-title'

# The font size to set for the title.
#
Header.Title.Font.Size = 'inherit'

# The font weight to set for the title.
#
Header.Title.Font.Weight = 'inherit'

# The font style to set for the title.
#
Header.Title.Font.Style = 'inherit'

# The ID assigned to the search terms.
#
Header.Terms.ID = 'pse-results-search-terms'

# The text color for the search terms
#
Header.Terms.Color = 'inherit'

# The font size to set for the search terms
#
Header.Terms.Font.Size = 'inherit'

# The font weight to set for the search terms
#
Header.Terms.Font.Weight = 'inherit'

# The font style to set for the search terms
#
Header.Terms.Font.Style = 'italic'

# Styling the previous / next links in the results overlay footer
#################################################################

# The ID assigned to the footer.
#
Footer.ID = 'pse-results-footer'

# The ID assigned to the previous page link.
#
Footer.PreviousPageLink.ID = 'pse-results-previous-page'

# The ID assigned to the next page link.
#
Footer.NextPageLink.ID = 'pse-results-next-page'

# The text color for the previous / next links
#
Footer.Link.Color = 'inherit'

# The text decoration for the previous / next links
#
Footer.Link.TextDecoration = 'none'

# The text color for the previous / next links when hovering
#
Footer.Link.Hover.Color = "inherit"

# The text decoration for the previous / next links when hovering
#
Footer.Link.Hover.TextDecoration = 'underline'

ResultItems.toml

Ditto for this file except that you'll see ClassName values popping up. Same deal as with those ID values I mentioned earlier. The file looks like this:

# Styling the result items section
##################################

# The ID assigned to the <section> element holding the results.
#
ID = 'pse-results-items'

# The ID assigned to the <ul> element.
#
List.ID = 'pse-results-list'

# Class assigned to item title links.
#
Item.ClassName = 'pse-result-item'

# Class assigned to an item's article element.
#
Item.Article.ClassName = 'pse-result-item-article'

# Styling the result item header
################################

# Class assigned to the header element within the article.
#
Item.Header.ClassName = 'pse-result-item-header'

# Class assigned to the item's title link.
#
Item.Title.ClassName = 'pse-result-item-title'

# The text color for item title links
#
Item.Title.Color = "inherit"

# The text decoration for the item title links
#
Item.Title.TextDecoration = 'inherit'

# the text color for item title links when hovering
#
Item.Title.Hover.Color = "inherit"

# The text decoration for the item title links when hovering
#
Item.Title.Hover.TextDecoration = 'inherit'

# The font size to set for item title links
#
Item.Title.Font.Size = 'inherit'

# The font weight to set for item title links
#
Item.Title.Font.Weight = 'inherit'

# The font style to set for item title links
#
Item.Title.Font.Style = 'inherit'

# Styling the result item body
#################################

# Class assigned to the <section> element with the result item body.
#
Item.Body.ClassName = 'pse-result-item-body'

# Class assigned to the item snippet.
#
Item.Snippet.ClassName = 'pse-result-item-snippet'

# The text color for result item snippets
#
Item.Snippet.Color = 'inherit'

# The font size for result item snippets
#
Item.Snippet.Font.Size = 'inherit'

# The font weight for result item snippets
#
Item.Snippet.Font.Weight = 'inherit'

# The font style for result item snippets
#
Item.Snippet.Font.Style = 'inherit'

# Class assigned to an item's thumbnail link.
#
Item.Thumbnail.ClassName = 'pse-result-item-thumbnail'

Alrighty, let's see how the stylesheet gets generated and hang 'em up for the day.

Style Compilation

Here's how all those parameter values populate the Sass file used to generate the CSS stylesheet (you can see why I quickly switched to block-based parameters in most of the other plugins):

div#{{ .SearchBar.ID }} { 
  position: relative;
  min-width: 1em;
  
  // The input
  input#{{ .SearchBar.Input.ID }} {    
    height: {{ .SearchBar.Input.Height }};
    transition: width {{ .SearchBar.Input.Transition.Duration }} {{ .SearchBar.Input.Transition.TimingFunction }};
    width: {{ .SearchBar.Input.Width }};
    color: {{ .SearchBar.Input.Color }};
    border-style: {{ .SearchBar.Input.Border.Style }};
    border-radius: {{ .SearchBar.Input.Border.Radius }};
    border-color: {{ .SearchBar.Input.Border.Color }};
    border-width: {{ .SearchBar.Input.Border.Width }};
    padding: {{ .SearchBar.Input.Padding.Y }} {{ .SearchBar.Input.Padding.X }};    
    outline-offset: -2px;
    -webkit-appearance: textfield;
          
    &::-webkit-search-cancel-button,
    &::-webkit-search-decoration {       
      -webkit-appearance: none;       
    }
    
    &::placeholder { color: {{ .SearchBar.Input.Placeholder.Color }}; }
          
  }
  
  
  // The button that toggles collapse state
  button#{{ .SearchBar.Button.ID }} {    
    background: transparent;
    border: none;
    color: {{ .SearchBar.Button.Color }};
    @include padding({{ .SearchBar.Button.Padding.Y }} {{ .SearchBar.Button.Padding.X }});
    margin: 0;
    z-index: 2;
    
    &[aria-expanded=true] {       
      position: absolute;
      right: 0.25em;
      top: 0.125em;
      color: black;    
    }
    
  }
  
}
  
// Search result wrapper
div#{{ .ResultsOverlay.ID }} {
  position: fixed;
  display: none;
  width: 100vw;
  height: 100vh;
  top: 0;
  left: 0;
  right: 0;
  bottom: 0;
  background: rgba(black, .5);
  z-index: 3;
  cursor: pointer;
  
  // Page results wrapper
  article#{{ .ResultsOverlay.Article.ID }} {  
    position: relative;
    overflow: auto;
   @include font-size({{ .ResultsOverlay.Article.Font.Size }});
    color: {{ .ResultsOverlay.Article.Color }};
    background-color: {{ .ResultsOverlay.Article.BG.Color }};
    display: flex;
    flex-direction: column;
    justify-content: stretch;
    align-items: stretch;
    width: 90vw;
    height: 90vh;
    left: 5vw;
    top: 5vh;
    
    @media (min-width: 769px) {
      width: 60vw;
      left: 20vw;
    }
    
    & > * { padding: 0; margin: 0; }

    // Page results header
    header#{{ .ResultsOverlay.Header.ID }} {      
      display: flex;
      justify-content: center;
      padding: 0;
      @include margin(1rem);
      color: {{ .ResultsOverlay.Header.Color }};
            
      #{{ .ResultsOverlay.Header.Title.ID }} {     
        padding: 0;
        margin: 0;
        @include font-size({{ .ResultsOverlay.Header.Title.Font.Size }});
        font-weight: {{ .ResultsOverlay.Header.Title.Font.Weight }};
        font-style: {{ .ResultsOverlay.Header.Title.Font.Style }};
        
        #{{ .ResultsOverlay.Header.Terms.ID }} {         
          color: {{ .ResultsOverlay.Header.Terms.Color }};
          @include font-size({{ .ResultsOverlay.Header.Terms.Font.Size }});
          font-weight: {{ .ResultsOverlay.Header.Terms.Font.Weight }};
          font-style: {{ .ResultsOverlay.Header.Terms.Font.Style }};
          
        }
        
      }
      
    }
    
    // Page results item section
    section#{{ .ResultItems.ID }} {
      padding: 0;
      margin: 0;
      flex-grow: 2;
      
      // Page results list
      ul#{{ .ResultItems.List.ID }} {       
        list-style: none;
        display: flex;
        flex-direction: column;
        align-items: stretch;
        gap: 1rem;
        padding: 0;
        @include margin(0 1rem);
        
        // Page results list item
        li.{{ .ResultItems.Item.ClassName }} {    
          padding: 0;
          margin: 0;
          position: relative;
          
          //Result content wrapper
          article.{{ .ResultItems.Item.Article.ClassName }} {         
            width: 100%;
            display: flex;
            flex-direction: column;
            align-items: stretch;
            gap: 0.5rem;
            padding: 0;
            margin: 0;            
            
            // Result header
            header.{{ .ResultItems.Item.Header.ClassName }} {      
              padding: 0;
              margin: 0;
              
              // Result title
              a.{{ .ResultItems.Item.Title.ClassName }} {
                color: {{ .ResultItems.Item.Title.Color }};
                text-decoration: {{ .ResultItems.Item.Title.TextDecoration }};
                @include font-size({{ .ResultItems.Item.Title.Font.Size }});
                font-weight: {{ .ResultItems.Item.Title.Font.Weight }};
                font-style: {{ .ResultItems.Item.Title.Font.Style }};
                
                &:hover {               
                  color: {{ .ResultItems.Item.Title.Hover.Color }};
                  text-decoration: {{ .ResultItems.Item.Title.Hover.TextDecoration }};
                  
                }
                
                &::after {
                  position: absolute;
                  top: 0;
                  right: 0;
                  bottom: 0;
                  left: 0;
                  z-index: 1;
                  content: "";
                }
              }
              
            }
            
            // Result body
            section.{{ .ResultItems.Item.Body.ClassName }} {    
              display: flex;
              flex-direction: column;
              align-items: flex-start;
              align-content: flex-start;
              gap: 1rem;
              
              @media (min-width: 576px) {           
                flex-direction: row;                
              }
              
              p.{{ .ResultItems.Item.Snippet.ClassName }} {           
                flex-grow: 1;
                color: {{ .ResultItems.Item.Snippet.Color }};
                @include font-size({{ .ResultItems.Item.Snippet.Font.Size }});
                font-weight: {{ .ResultItems.Item.Snippet.Font.Weight }};
                font-style: {{ .ResultItems.Item.Snippet.Font.Style }};
                
              }
              
              a.{{ .ResultItems.Item.Thumbnail.ClassName }} { }
              
            }
            
          }
          
        }
        
      }
      
    }
    
    footer#{{ .ResultsOverlay.Footer.ID }} {
      margin: 1rem;
      display: grid;
      grid-template-columns: 1fr auto 1fr;
      grid-template-areas: "previous . next";
      place-items: center;
      
      a {        
        background: transparent;
        color: {{ .ResultsOverlay.Footer.Link.Color }};
        text-decoration: {{ .ResultsOverlay.Footer.Link.TextDecoration }};
        
        &:hover {           
          color: {{ .ResultsOverlay.Footer.Link.Hover.Color }};
          text-decoration: {{ .ResultsOverlay.Footer.Link.Hover.TextDecoration }};
          
        }
        
        &#{{ .ResultsOverlay.Footer.PreviousPageLink.ID }} {          
          grid-area: previous;
          align-self: start;
          
        }
        
        &#{{ .ResultsOverlay.Footer.NextPageLink.ID }} {          
          grid-area: next;
          align-self: end;
          
        }
        
      }
      
    }
    
  }
  
}

Alright. Y'all good? I'm good. If y'all aren't good, just holler.