Friday, September 30, 2016

Reading 8 - ShortStraw

Paper
Wolin, A., Eoff, B., & Hammond, T. (2008, June). ShortStraw: A Simple and Effective Corner Finder for Polylines. In SBM (pp. 33-40).

Notes
Continuing on corner finding, we will now look at a very simple segmentation algorithm -- ShortStraw

Wednesday, September 28, 2016

Reading 7 - Sezgin and Stahovich

Paper
Sezgin, Tevfik Metin, Thomas Stahovich, and Randall Davis. "Sketch based interfaces: early processing for sketch understanding." ACM SIGGRAPH 2006 Courses. ACM, 2006.

Notes
Up to this point, we've been concerned with understanding and building features of strokes and using these features to describe and classify gestures.  With that background on machine learning and the familiarity gained with working on sketch data, we are now moving into more sketch-specific problems.  To begin with, we will look at corner detection for a few readings.  One of the more important original works in corner detection also helps to communicate the intuition behind the features that are often used, so we will be starting with the Sezgin/Stahovich paper shared in the class drive.

Wednesday, September 21, 2016

Reading 6 - Geometry Overview

Paper
Hammond, T. "Fundamentals of Geometry". Draft

Notes
In the shared folder for Readings, you can access the draft version of Dr. Hammond's paper on basic geometry.  This paper provides some background information on dealing with geometries, vector spaces, and matrices.  Not only is the knowledge applicable to a number of classification methods, such as our current linear classification discussion, but it is also a good review of trigonometry and functions that we often deal with in feature extraction.  As we move on to other methods of recognition, this intuition will be useful.

Sunday, September 18, 2016

Reading 5 - Metrics Overview

Paper
Hammond, T. "Evalutation Methods and Metrics". Draft

Notes
In the shared folder for Readings, you can access the draft version of Dr. Hammond's paper on evaluation metrics.  This paper is a general overview of some common methods used in evaluation and accuracy reporting for machine learning problems.  In this class, we'll be dealing with features and classifiers in lectures, homeworks, and more, so having a good understanding of the basics of evaluation will be very helpful.

Tuesday, September 13, 2016

Homework 1 - Features and Classification

Overview
This marks the first programming-based homework assignment of the course.  The intention behind this assignment is to prepare you for the class project and give some practical experience using the material we've been discussing in lecture.  You will need to implement feature transformations and perform some classification on those features for this homework.  It may be considered across two parts:

1. Rubine Features
2. Weka Classification

Instructions
There is no required programming language for this homework.  The only requirements are that you implement Rubine and use Weka.  To measure your performance on these tasks, you will need to follow some data format guidelines -- use the data provided and make sure your feature extractor outputs or can be made to easily output a CSV file.  These details will be discussed shortly, but keep them in mind when choosing how you wish to begin the homework.

Also, a "Homework 1" directory has been added inside the shared class Google Drive.  It contains all the downloadable materials for this assignment.

1. Rubine Features

a. Data

First, you'll need the data.  There are two data sets.  One is the small sample set which we have seen in quizzes and in discussion.  The sample CSV output is also available for this data set so that you may check your feature extractor before applying it to the second data set.  The second data set is a collection of alphabet letters.  There are 20 samples for each letter.

Data is available to you in three ways.

i. TXT Files

The TXT version of the data is the most rudimentary means of storing a sketch available.  These files list only the points, with each line containing x, y, and t for a single point.  Reading the raw data from these files should be relatively simple, but there's no inherent structure about the sketch that is saved.

ii. JSON files

The JSON version of the data is saved in the SketchML::JSON format, developed by the Sketch Recognition Lab and based on MIT's original SketchML format on the XML platform.  This format is still fairly lightweight compared to XML, although not as compact as the TXT format.  It provides structure to the data, giving a collection of points, strokes, and more to the sketch object.  JSON is very easy to work with in many modern languages, but you may wish to reference the SketchML::JSON specification document if you decide to parse the JSON yourself.  It is saved alongside the data in the same folder.

iii. Sketch Recognition Library API

The Sketch Recognition Library (srlib) is a collection of sketches from different domains that have been gathered into a single format (SketchML::JSON) and made available through a RESTful API.  All the data used for this assignment may be accessed from the srlib API, but it is not available on the Google Drive.  It is live online from http://srl-prod1.cs.tamu.edu:7750/.  You must be on campus or on the VPN to access it.  That link will direct you to the documentation.  All you need to get started is that the "sample" data set is available with the domain "rubine" and the "letters" data set is available with the domain "letters".  An example is included in the starter code discussed below.

b. Feature Extraction

Regardless of how you choose to get the data, you must implement all 13 of Rubine's features.  The sample data set is provided to assist you in debugging, while the larger data set of letters will be used in the next stage with Weka.

To provide further assistance with this step, starter code has been included in the shared folder.  The starter code does not implement any of the feature transforms.  It is actually intended to be a viewer for you as another means of debugging.  To use it as a viewer, download all the files in the starter folder and open "index.html" in your browser.

Because it uses the srlib database and includes a special build of the srlib Javascript toolkit (data management functions only), it is essentially the same code you would be writing to access the data from srlib in Javascript on your own.  For that reason, I added a single line, a callback function to an empty "getFeatures()" function where you may implement your feature extractor if you wish to work with the starter code.  Again, you may use any language you want, so there's no requirement to use the starter code.  Even if you do wish to work in Javascript, you may want to look at the data in its raw TXT or JSON form to see how it looks.  But because this homework is concerned with feature extraction and classification, not data handling and plotting, the viewer and Javascript tools are available as a resource.

c. Output

The reason the language doesn't matter is because the output will be important.  Your program should generate a CSV file, or log console output which can be readily saved as CSV, where each row represents a sketch.  Thus, for the sample data set, you will have 8 rows of features.  The letters data set will have 20*26 rows.

Each row should contain the sketch's class/interpretation followed by the 13 Rubine feature values in order of F1 through F13.  For the sample data set, the class can just be the name or ID; it doesn't really matter.  For the letters data set, you should save the letter as the class.  The letter is saved in the SketchML::JSON data under the top-level shape's "interpretation" field.

The file "sample.csv" provides the Rubine feature values for the sample data set to assist in your evaluation.  You will need to generate "letters.csv" using your feature extractor.

2. Weka Classification

The second half of the assignment will not require any additional programming.  For this part, you'll be using Weka to build some classifiers with the features you extracted in the previous part.  Weka can import CSV directly, which is one of the primary reasons that your feature extractor should support saving CSV features... the other being ease of grading.

Once you have a "letters.csv" which has the class label and set of 13 features for all 520 sketches, import the data into the Weka Explorer.  From Explorer mode, you can test out many different functions in Weka, including classification, dimensionality reduction, and visualization.

Try at least 5 different classifiers on the data.  Think about mixing k-fold cross-validation and other data splitting methods.  You should report the classifiers and settings you chose along with the results in your report to be submitted with the code and CSV files.

I've tried to include all the materials you'll need either online or in the shared class folder.

At the end of both parts of the homework, you'll have gained some familiarity with gesture-based sketch features and classification methods through Weka.  Later, we'll be investigating the programming-based recognition a bit more, so you will probably have the chance to implement classifiers, segmenters, and other sketch algorithms in upcoming assignments.

Obtaining Credit
You will need to email the grader all of your files.

For simplicity, place all your files inside a single folder named "HW1_<last name>_<first name>" that will be compressed and submitted as a ZIP.  You should include the following files:

+ If you decided to download a data set (JSON or TXT), include these files in a "data" directory.

+ Include the source code which reads the data and generates the features in a "source" directory.  This can be a single file or multiple files, even a combination of scripts.  If your source is more than a single file, please add a small README that tells which one to run.

+ Include your CSV files for both the sample and letters data sets in a "results" directory.

+ Include the output of the weka console after testing each of your 5 different classifiers and configurations in the same "results" directory with the CSV files.  This can just be a TXT file that you make by copying-and-pasting from the weka window.

+ Include your report which (briefly) discusses the options you tried out in Weka and reports your findings in the top level.  You should have at least a small discussion of your findings.  Were you pleased with the results?  Did the exercise help you think of different features which you feel would be more important?  Any general impressions you had during either part of the assignment could be excellent material for a discussion.  (e.g. "First, I tried a Random Forest with 10-fold cross validation and obtained an F-measure of 86.4%.  Next, I wanted to see how a neural network would perform, so I ran a multilayer perceptron with...")

To recap, a submission might look like this:

HW1_Polsley_Seth/
|-- data/
|---- sample-json/ (contains all the sample json files)
|---- letters-json/ (contains all the letter json files)
|-- source/
|---- rubine.py
|-- results/
|---- sample.csv
|---- letters.csv
|---- wekalog.txt
|-- report.txt

This layout isn't an exact science.  I mainly want to make sure you include everything to demonstrate that you completed each part, and the structure is intended to make it easier for you to show that.  If you used the Sketch Recognition Library starter code, you can have many fewer files.  If you wish to consolidate the weka log into your report, that is also fine.  So another submission may be:

HW1_Polsley_Seth/
|-- index.html
|-- require.js
|-- srlib.js
|-- sample.csv
|-- letters.csv
|-- report.pdf

This is also fine.  I mostly just need your code, the code output (csv files), weka output (weka log), and a short write-up (discussion/report).  Include these all in some obvious manner, and it's ok.

Again, please ZIP all your files together into a single file submission titled "HW1_<last name>_<first initial>.zip"

As always, contact me if you have any questions.

Due Date
Sep. 26, Monday @ Midnight
25% deducted per day late

Monday, September 12, 2016

Reading 4 - Gesture Overview

Paper
Hammond, T. "Introduction to Gesture Recognition". Draft

Notes
In the shared folder for Readings, you can access the draft version of Dr. Hammond's paper on gesture recognition.  It is an overview of all the previous gesture work we've discussed and provides many practical explanations and examples to help cement these concepts before we move on into classification techniques and other types of recognizers.

Sunday, September 11, 2016

Reading 3 - Long

Paper
Long Jr, A. Chris, et al. "Visual similarity of pen gestures." Proceedings of the SIGCHI conference on Human Factors in Computing Systems. ACM, 2000.

Notes
Long's analysis of gesture features builds further on Rubine's original work in the field.  The most important section of this paper is the discussion and results of the set of experiments measuring the most useful features of gesture similarity.  As a small reminder, while the Monday and Wednesday readings should have summary blog posts completed by this weekend, this reading and other Friday readings are not due until the next weekend.  The Reading Policy post has more details.

Wednesday, September 7, 2016

Assignment 6 - Project Exploration

Overview
Following up on Assignment 5, you will now do some very tentative exploration into the project area which is interesting to you.  This should be a small assignment; the intention is to start helping you build a little background experience in the domain and reshape your ideas based on the prior work.

Instructions
Go to Google Scholar and input some general search terms related to your project idea(s).  Based on just the titles and abstracts, find 10 prior works which appear to be relevant to your own project goals.  You do not have to read these papers, simply find 10 that appear to be related.  You should then create a blog post with the citations of all 10 papers.  Hint: Clicking "Cite" under a result in Google Scholar will give you the citation information; MLA is sufficient for listing on your blog.

Obtaining Credit
You should have a blog post corresponding to Assignment 6 posted on your blog by the due date.  Again, this is simply a list of 10 paper citations related to your project idea(s).

Due Date
Sep. 12, Monday @ 11:30 AM
25% deducted per day late

Reading Policy

Overview
As promised earlier, I wanted to provide a little more detail about the readings in this course.  I've given more information below in regards to the reading assignment schedule, in-class deliverables like quizzes, and out-of-class deliverables like blog posts.

Readings Assignment Schedule
Readings will be regularly assigned in class M, W, and F.  There won't necessarily be a reading every day, but they'll roughly correspond to this schedule.  They are not meant to be an onerous burden; rather, readings are given to provide you some background and supplemental material for the content of the class.

In-Class Deliverables
In-class discussions and quiz questions may draw on material from the readings.  Generally, you should review a reading before the next class from which it was assigned.  Thus, a reading assigned Monday might be discussed on Wednesday.  A reading given on Friday would be fair game for a quiz question on Monday.  I say to "review" readings because you are encouraged to go through readings at whatever pace is comfortable for you, so that you understand the concept and key points.  You do not have to do a detailed, rigorous review.

Out-of-Class Deliverables
Outside of coverage of readings in discussion and quizzes, the primary graded component to a reading will be a blog posts.  Using the blog you created in Assignment 2, you'll be posting a little bit about each reading.

Blog Post Contents
You should post the name and reference of the paper concerned, a short summary, and any thoughts you had about the paper.  These posts are primarily for your benefit to use as reference later when working on your project paper or reviewing readings for quizzes.  They do not need to be long; just communicate what you took away from the paper.

Blog Grading Timeline
Rather than check everything at the end of the class, I think it will be easier for everybody to do regular checks.  This will ensure that you are keeping up in the course without my constantly checking everyone's blogs.

To that effect, I plan to be looking at everyone's blogs once a week on Sunday afternoons.  While in-class deliverables may occur as early as the next class, you will always have at least half a week for out-of-class deliverables.  Thus, the blog posts for readings assigned on Mondays and Wednesdays will be due by that upcoming Sunday.  The blog posts for readings assigned on Friday will be due the next Sunday.  Consider the following example:


 Su Mo Tu We Th Fr Sa 
              1  2  3 
  4  5  6  7  8  9 10 
 11 12 13 14 15 16 17 
 18 19 20 21 22 23 24 
 25 26 27 28 29 30    

The calendar above shows the month of September and sample reading assignments.  All the red readings should have a corresponding blog post by the bold-red Sunday.  Likewise, all the blue readings should have a corresponding blog post by the bold-blue Sunday, and so on.  This gives you a little over a week to finish Friday readings, slightly under a week to finish Monday readings, and half a week to finish Wednesday readings.  A concrete example: Readings 1 and 2 will need to be blogged by this coming Sunday, Sept. 11.  If there is a Reading 3 on Friday, it would not be due until Sunday, Sept. 18.

Note that the time I'll be checking on Sunday is intentionally left vague.  This is because I really intend for everything to be posted by the end of Saturday, but I won't be checking that night.  You can update on Sunday afternoon, but there's a possibility I'll have already checked your blog, so try to be a little earlier rather than later.  There won't be any late submissions for readings, but since there are likely to be a lot of readings, missing a few will not have a very negative impact.

Conclusions
The key takeaways from this post:

  • Readings will be given on class days
  • They may be covered in discussion or quizzes as early as the next class, so a small review by then is encouraged
  • The blog posts, which are really just your summaries of the readings, will be due the upcoming Sunday, except Friday readings will be due the next Sunday
  • The grading time on Sunday is intentionally-ambiguous so that you may aim to have everything done by Saturday
  • There is no late policy for readings, but don't worry about a missing a few here or there; a few will be dropped at the end as means of grading leniency in favor of late submissions
Email me if you have any questions or concerns regarding this policy.

Reading 2 - Rubine

Paper
Rubine, Dean. Specifying gestures by example. Vol. 25. No. 4. ACM, 1991.

Notes
Here we begin to study the origins of modern sketch recognition from an algorithmic perspective.  The most important section to review from this work is part 4, which covers Rubine's features.  Rubine's features partially form the basis of many gesture-based recognizers today.  At the very least, familiarity with these sorts of feature constructs will help you gain an intuition for the type of information we can capture and use about a sketch.

Monday, September 5, 2016

Reading 1 - Dialectical Creativity

Paper
Hammond, Tracy. "Dialectical Creativity: Sketch-Negate-Create." Studying Visual and Spatial Reasoning for Design Creativity. Springer Netherlands, 2015. 91-108.

Notes
This is a good introductory reading in the field of general sketch recognition.  It should help motivate upcoming content in the course and encourage project ideas.

You may access the paper from the Google drive folder "CSCE624 Class Files" available to all students in the class.  If you did not receive access, let me know so you may be added to the class group.

Assignment 5 - Project Ideas

Overview
Even though it's only the second week of class, it will be helpful to start thinking about what aspects of sketch recognition interest you and, more specifically, shaping these ideas into potential topics for your end-of-semester project.  To further that goal, this assignment is that you post a paragraph about potential topic ideas to your blog.

Instructions
Post a paragraph on your blog regarding potential topic ideas for the class project.  It does not have to be very refined yet.  Think about your interests; why are you interested to learn sketch recognition?  Can this be morphed into a project?  Perhaps one of the ideas already presented in class is appealing to you, or your own research holds a sub-topic related to sketch recognition.  Again, the purpose of this assignment is to encourage you to start thinking about the final project, so spend some time brainstorming, and write up your best idea(s).

Obtaining Credit
You should have a blog post corresponding to Assignment 5 posted on your blog by the due date.

Due Date
Sep. 7, Wednesday @ 11:30 AM
25% deducted per day late
It is a very simple exercise primarily for your own benefit, so everyone should be able to complete it by class on Wednesday.

Saturday, September 3, 2016

Assignment 4 - Persketchtivity Usage and Analysis

Overview
Following on the previous assignment in which you completed a tutorial in Mechanix, you will be using another live educational sketching tool: Persketchtivity.  This software should be much easier to use because it is available online, has automatic lesson progression, and only consists of drawing simple shapes.  After using Persketchtivity, you will be asked to once again write about your impressions in a list of "Do's and Don'ts of Sketch Recognition Systems, Part 2".

Instructions
First, you should complete the steps below to gain some experience using Persketchtivity.
  1. Go to http://persketchtivity.com/ to access the main Persketchtivity log in page.
  2. Return to the User ID spreadsheet that you used to log in to Mechanix.  Your user id is the same for Persketchtivity.  Once again, note that the password and user name are equal.
  3. Complete the available lessons.  This consists of four different primitive shape lessons and a few 3D shapes and combos.  Many of the lessons have sub modules, but all of the exercises are small and should not take much time to complete.
Next, you must write a short post on your blog about the "Do's and Don'ts of Sketch Recognition Systems, Part 2".  Specifically, list at least 5 aspects of Persketchtivity which you liked as the "Do's" and list at least 5 aspects of Persketchtivity which you disliked as the "Don'ts".  These aspects may refer to the usability of the system, design of the interface, or any other observation relevant to sketch recognition software in some way.  You may present the lists as bullets or paragraphs, but there should be 10 distinct observations clearly contributed in the post.

Obtaining Credit
Your submissions will automatically be visible to us on the instructor side, so you do not need to submit any certificate or problem solutions to show that you completed the lessons.  However, we need to know who is who, so you must use the user name provided in the spreadsheet.  Remember the user name and passwords are the same.

To obtain credit for the "Do's and Don'ts of Sketch Recognition Systems, Part 2", ensure you have posted your 10 observations (5 do's and 5 don'ts) to your blog.  This should be the same blog you provided a link to on the spreadsheet provided with Assignment 2.

Due Date
This assignment should be submitted in two parts.  It should be completely finished by Monday Sep. 5, but the Persketchtivity exercises should be done by class on Monday Sep. 2.

Persketchtivity Exercises
Sep. 5, Monday @ 11:30 AM
Only excused late submissions (e.g. late enrollment)
Consider this as a sort of "checkpoint" so that we can ensure everyone has gained familiarity with Persketchtivity and address any problems that arise before the blog assignment is due that night.

"Do's and Don'ts of Sketch Recognition Systems, Part 2" Blog Post
Sep. 5, Monday @ Midnight
25% deducted per day late
This is when the second "Do's and Don'ts" list should be posted on your blog.