Apperceive.net

Documentation

This is an experimental application that arose out of a desire to more efficiently search newspaper issue archives and visually scan large results sets of the Palacios Beacon though UT The Portal to Texas History. 

Initial use cases envisioned

- Browse front pages between two dates without having to navigate from results page to standard view page, then to zoom page, and then back to results page and on to next item (or, alternatively, opening each result in a separate tab and moving through standard and zoom pages). 
- When finding a result that one is looking for, one often has to navigate to other pages for metadata or OCR scan.
- For a given set of dates, find the first issue published after that date. 

Motivation was to 

- provide alternative to Portal date filtering (not just decades, years, months, etc). This allows a start and end date
- significantly reduce clicks required to visually scan results by drilling down (clicking) 
- combine data typically found of different Portal pages (e.g., issue page count and other item metadata), again reducing clicks
- turn searches that take hours into minutes
- make complex searches easier to manage and store
- turn results into content

Modes of operation

- there are two tabs: date range and date list
- the first, date range, one allows searching within a date range
- the second, date list, finds nearest issue after each date (example, show all front pages from issues after know hurricane dates)

Features

- jump to result issue using selection, slider and thumbnails
- links to common pages (standard, zoom, all pages)
- show count of pages in each issue 
- links to sample date range data (Texas Hurricanes and Markus' birthdays)
- date list takes a comma separated list of dates in the format YYYY-MM-DD
- rapid review and result drill down
- swipe based navigation from result to result
- thumbnails and large images

Critical bugs

- FIXED 2018/07/26.  Date range calendar dates in the URL query-string and date range inputs ignored and defaults used.

Known bugs

- does not accept keywords yet
- only displays front page at this time
- not tested outside chrome for windows
- calendar date selection is not intuitive. For example, changing year is not enough, user must also click a day in pop up calendar else date is not changed.
- depending on cursor position, mouse scroll wheel navigates between items or scrolls page. A feature to some, a bug to others. Allow setting for preferences?

Wish list

- put this all on github, set up dev/test/prod versions
- demo video
- auto calculate date list for a person's lifetime of birthdays. See all issues printed within a week of a person's birthdays.
- allow for after, before and nearest item date matches in date list mode
- test on tablets/mobile smaller screens, test touch screens
- test in Firefox, safari, IE
- evaluate more ways to make this "lazy"
- evaluate more ways to "preprocess". Currently a master list of all Palacios Beacon issue dates and unique ID's (metapths) for items at the Portal is cached and used to facilitate date range and list searches
- look into generating a date/item lookup table for any Portal Collection
- change dates list examples from links to auto-fill of dates list input field
- allow tagging of specific results with keywords or notes
- allow "remove item" from results to allow fine tuning results
- allow "add" item to combine multiple queries (useful when searching multiple words/phrases, one at a time). 
- A remove duplicates feature may be useful when more complex queries are allowed.
- output results to a pdf or other useful formats 
- make all this work with open library api that Portal and others are using.
- will this work with newspapers.com?
- talk with tech/IT folks at Portal about this
- save preference settings (cookies or full blown user accounts?)
- integrate into a WordPress or Drupal site where people can login, save "projects" containing keywords, dates, and "curated" results that can be published and shared on-line. add user accounts to facilitate saves and multiple project "folders"
- save to time-line, use knight data story telling 
- save to video?
- save as screen saver (just a folder?)
- better page layout and design
- allow for "slide-show" with resulting pages in browser. full screen? set time for transitions?
- ensure that 0 results is not a bug, same for bad dates in parameters (both same or reversed)
sanitize all input dates, bind parameters
- allow search phrase(s), apply against item to see if there are hits? or use ocr? indicate when one or more search phrases are found on front page or in issue
- make short, weekly videos of "this week in Palacios history" with ~100*60% front pages, perhaps curate to 10?  or use only material that is rights free, so < 1924
- be more forgiving with inputing dates and formating, also messages to help find bad user data
- limit searches to specific pages (e.g., front page or 1-3) by using page ocr's or other method (search inside and look at which pages matches occur on)
- print all or download all images easily
- refresh all date to metapth data when new additions to portal collections. how tell easy if new items?
- print or export/zip everything/anything 
- use placeholder examples and tool-tips on input fields
- cache images for a limited time
- cleaner way to go from date range to date list regarding URL querystring but don't lose data?  use a hidden for mode?
- consolidate repeated database queries to reduce load - reset rs pointer?
- settings tab
- use hidden to allow no search when page appears
- allow labels in date list csv YYYY/MM/DD space Label text comma. show labels with issue metadata

Additional Notes

- see original developer notes and source code for additional comments
- most of this is just simple PHP, open API's, javascript, jQuery, mysql, swiper javascript library and Bootstrap 3.
- note: searching on front page does not tend to have problem with "this day in history redundancies"
- see portal3.php for database update code for date to metapth lookup. now stores additional metadata.
- portal2.php is original, text only, simple experimental prototype
- trivial timeline experiments in old/timeline.php
- early output extracts in colored_school.html, negro_school.html, mexican_school.html in folder "old"

Information

- Apperceive.net is a project of Apperceive.com
- for more information, contact markus.sandy@gmail.com, 361-532-9596