Updated date:

How to Make Better Machine Learning Typing Gloves

I am an Information Technology degree holder. I recently earned three certificates in Tiny Machine Learning.

how-to-make-machine-learning-typing-gloves-that-are-better-than-mine

This Tutorial Will Teach You How to Make Your Own ML Gloves

This tutorial will walk you through a project I made in my spare time using some electronic components I had lying around and knowledge of Python, Arduino, and ML. The project focuses on pioneering a very niche and unique area where the three former disciplines I mentioned intersect. As you may have guessed from the title, this isn't a tutorial on making the perfect Machine Learning Typing Gloves.

This tutorial will give you some ideas on how to start making something that works pretty well. This could mean that the final accuracy of your model is low or that the gloves only work some of the time.

There are also three other modules besides pythonserialmlgloves1.1, including the mlgloves1.1 containing the machine learning model and Arduino code, so that this tutorial will be separated into a series of three tutorials. If you are willing to delve into this topic in greater detail, then without further delay.

Here is how to make the gloves.

Required Items

10 Flex Sensors (made from straws photo-resistors and resistors)

2 Arduino Lilypads (or similar product, but keep in mind the voltage on the output pins and the number of analog pins)

2 Bluetooth modules for Arduino (I used the Silver Mate from sparkfun.com)

2 AAA battery cells

1 PC (powerful enough to do the Arduino and Python programming on)

1 PC Bluetooth module

1 Electrical tape (optional)

Various colored wires of different lengths

Some Words Of Caution Before You Begin

Before we start, I'd like to talk about some problems I ran into during this project and things that I could have done differently.

The primary thing that wasn't obvious to me at the time was something I'd only learn about later after I started this project. There is a new field of Machine Learning that is just beginning to take off called Tiny ML. I first learned of this under the name edge computing. It is a very new way of using Machine Learning on small power-constrained devices.

The drawback of using doing ML the traditional method in something like TensorFlow 1.x was that in the end, the inference had to be done on the computer. If the gloves had become prevalent, it would have meant that the device wouldn't have been very portable. I would have instead had the inference done on the Arduino Lilypads, so I could have made an app on Android or IOS that would receive the ML inference from the gloves.

Another drawback of the gloves is that in the end, they were only able to recognize about ten characters from the keyboard reliably. I chalked this up to simply not having enough training data to support a fully functioning model. Still, I have to admit that seeing it work at all after the many long hours I spent on the project was satisfying enough.

The last problem I ran into was the flex sensors not being sensitive enough to catch all of the motions of the fingers, as hardly any resistance changes for some keys typed. Putting the flex sensors contained within the straws could remedy this.

Also, during the writing of this article, I was advised that the framing of it was necessary. That I should not use the F word. No, not that F word, but the word failure. So here on out, problems with the project will be referred to as drawbacks.

how-to-make-machine-learning-typing-gloves-that-are-better-than-mine

1. Making the Flex Sensors

This project requires the use of flex sensors to record the finger movements of the hands when typing the data to train the Machine Learning Model.

The flex sensors I used were DIY, they work by a light shone against a photoresistor. The bending of the fingers changes the light let through the straw onto the photoresistor.

Materials Required

10 Photoresistors

10 LEDS(Light Emitting Diode)

10 470 ohm resistors
10 10 Kohm resistors

Copper wire of varying lengths
Insulating tape or hot glue to prevent short-circuits or conceal the light
10 plastic straw
Soldering iron
Boost Converter or 9-volt cell source
Solder

Instructions

Step 1: The straw will have to be cut to the right length. This measurement could vary depending on the length of the fingers, but this will likely be several inches. The pinky, thumb, and middle fingers will likely be relatively different sizes, so will need different straw lengths.

Step 2: The LED should be connected to both sides of the flex sensor circuit. This will be at the opposing end from where you are going to put the photoresistor. Be sure you know which side of the LED is positive and which side is negative. For remembering this, I have a mnemonic that goes like Annual Polkadot Conference Nexus. Anode->Positive->Cathode->Negative.

Step 3: The positive side will be connected to a 470-ohm resistor. Between the 470 ohm resistor and a 10K ohm resistor, a wire needs to be soldered. This is so the LED will receive approximately 19 milliamps of current, while the photoresistor gets a relatively small amount of current. This wire will either go to a boosted 9v source, or an extra power source.

Step 4: The photoresistor should be placed on the opposite side of the flex sensor circuit(where the LED is). One end of the photoresistor will be connected to one of the analog pins (A0, A1, A2, A3, A4) for each finger. The other end will be connected to the ground and the other side of the straw, up to the LED.

Step 5: (drawback )It is important to remember at this point that once you have all the components put together the flex sensor will probably work better if all the components are within the straw.

Step 6: The power source for the flex sensor can come from a separate 9-volt cell. But I suggest trying to use a boost converter to boost 3.3 volts or 5 v to 9v, to have enough power for the LEDs.

Step 7: The pull-down resistors may need to be used, but you'll have to look into this.

2. Getting Into the Analysis of the Project

It makes the most sense to start with the harvesting of data, which will build the foundation for the rest of the project. Essentially what pyhthonserialmlgloves1.1.py does, is capture data in the form of a command-line interface.

The user training the gloves would type a certain amount of times to have enough data for the model to learn the associations to make inferences in the ML model script. We communicate all the data from the Arduino Lilypads (or similar Arduino devices) through the serial communication ports.

Each Bluetooth module should have its com port. The data is then extracted and processed, then exported to a CSV. By performing this action, it allows the ML model script, which a future tutorial will explain, to perform inference(classification).

The following code imports a few things so that for Python to use. We now import the library serial, as we'll be working with serial communication to and from the Arduino Lilypads. We also import re to do regular expressions. After this, we import threading because it's essential not to have parts of the program block.

Blocking methods in Python block the flow of the program from continuing. serial and msvcrt methods are blocking. Therefore, it is vital to use the serial and critical capturing parts of the program to keep this in mind.

Finally, we import msvcrt and CSV for key capturing and working with comma-separated values.

Required Libraries

import serial
import re
import threading
import msvcrt
import csv

Serial in the Morning

The code found below opens up serial communication channels between the computer and the Arduinos. The variable ser0 is for one hand while ser1 is for the other.

The arguments given are for setting which com ports will be opened, things like baud rate, and also how much data will be transferred.

# This was changed to account for the newly added analog sensors on each hand.
# Setting it to 23 bytes was cutting off the analog sensor #9
#HC-06(RED)
ser0 = serial.Serial("COM9", 1200,bytesize=serial.EIGHTBITS,timeout=None, parity=serial.PARITY_NONE, rtscts=1)
#RN(GREEN)
ser1 = serial.Serial("COM10", 1200,bytesize=serial.EIGHTBITS,timeout=None, parity=serial.PARITY_NONE, rtscts=1)
# Class to do the important procedures on the serial data

Having Some Class

This is the class serprocedure. It does most of the processing for the data coming into the computer.

First, we declare our class, then we try them to see if the properties do not already exist. If they don't exist then we declare/initialize them.

# Class to do the important procedures on the serial data
class serprocedure():
	# The following variables are class-level and are static, which means they will persist across class instances
	try:
		sensorlist
	except NameError:
		sensorlist = []
	# A list to store the missing data
	try:
		missingdata
	except NameError:
		missingdata = []
	# A counter for missing data
	try:
		mcounter
	except NameError:
		mcounter = 0
	# Boolean to store true false as to whether serdatalooper gets all data on the first run
	try:
		First_Try
	except NameError:
		First_Try = False
	try:
		times_counter
	except NameError:
		times_counter = 0

Being Constructive

The init constructor is then defined. We use flags to control the flow of the program and decide whether or not to perform further extraction or to throw away the result.

		# Use the __init__ constructor to take argument self and serdata
		# Each method should only do 1 thing
		# Use the __init__ constructor to take argument self and serdata
		# Each method should only do 1 thing
		def __init__(self,serdata,flag,key):
			self.serdata = serdata
			self.flag = flag
			self.serdatalooper(self.flag)
			self.key = key
			# If it is the second thread with the second serial com, and missing counter less than 1, and sensorlist not greater than 10
			if self.flag == 1 and serprocedure.mcounter < 1 and len(serprocedure.sensorlist) == 10:
				# Tell the user that the program is performing further extraction
				print("Performing further extraction of data and exportation")
				# Perform further extraction
				self.furtherextraction(serprocedure.sensorlist)
				serprocedure.times_counter = serprocedure.times_counter + 1
				# This tells how many times the user has typed 
				print("You've typed " + str(serprocedure.times_counter) + " Time(s)" ". You just typed the letter " + str(self.key))
			elif self.flag != 0:
				# Reset counter
				serprocedure.mcounter = 0
				# Clear the list so it doesn't build up
				serprocedure.sensorlist.clear()
				print("Throwing away partial result.")
			# Check if there is missing data, if there is loop through that. Otherwise loop through the data normally.

Doing Your Research

If you are familiar with regular expressions, then you'll be at home with re.Search. Re.search searches through the data for the analog indicator and some digits. We perform validation to check if found does not equal None.

After this, if we find a group, then it appends to the sensor list. If we see this, the mcounter also is incremented by 1.

		# Method to extract the individual parts of the serial data
		def extractanalog(self,analognumber,flag):
			# Changed the decimal regexp to {1,2} quantifier
			found = re.search(r'(A' + str(analognumber) + '\d{1,2})',str(self.serdata))
			if found is not None:
				if found.group():
					# Create a list of data
					# sensorlist must be moved to the top to be a class-level variable
					#sensorlist = []
					serprocedure.sensorlist.append(str(found.group()))
					return
			else:
				serprocedure.mcounter += 1
				# It's getting stuck here
				return

Going Further

Further extraction performs further data processing and extraction on the data coming in from the ML gloves.

		def furtherextraction(self,newlist):
			# A list to hold analog labels
			findanaloglabel = []
			# A list to hold analog values
			findanalogvalue = []
			z = 0
			print("This is the list in furtherextraction:")
			print(newlist)
			# Len counts 10 elements in the list but the index starts at 0
			while z < len(newlist):
				# These will have to be made into lists
				findanaloglabel.append(re.search(r'(A\d)',newlist[z]).group())
				# ?<= looks behind and only matches whats ahead
				# Changed the decimal regexp to {1,2} quantifier
				findanalogvalue.append(re.search(r'((?<=A\d{1})\d{1,2})',newlist[z]).group())
				# Increment z
				z = z + 1
				# Call the export method
			self.exporttocsv(findanaloglabel,findanalogvalue)
		# Export to excel form

Together but Apart

Exporttocsv is a method used to take all of the gathered data from further extraction and the code in extract analog and puts them into neat individual cells.

		def exporttocsv(self,labels,values):
			# Insert key value into list
			values.insert(0,self.key)
			with open('directory','a',newline='') as csvalues:
				fhandle = csv.writer(csvalues)
				# Export the row to csv file
				fhandle.writerow(values)
				print("Done exporting values to csv")
				if self.flag == 1:
					serprocedure.sensorlist.clear()

The Bigger the Loop, the Bigger the Programmer

Serdatalooper loops through values for each flex sensor. If the flag is set to 0 it is one hand, otherwise, it is the other hand's data.

		def serdatalooper(self,flag):
			if flag == 0:
				i = 0
				end = i + 4
			else:
				i = 5
				end = i + 4
			# Loop through the entire length of the list and extract the data
			while i <= end:
				self.extractanalog(i,"mainlist")
				# Increment the counter
				i = i + 1
				# Sort the list
			serprocedure.sensorlist.sort()
				
			#if len(serprocedure.missingdata) < 1:
				#q.put("There were no missing data")
				#return True
			#else:
				#q.put("There are " + str(len(serprocedure.missingdata)) + " of data missing from list")
				#return False

3. Another Example, Please!

When I did this, I probably could have gathered more data than I did.

If you're trying to improve on the design of the gloves, aiming for a dataset with 1000 examples per character might be a good goal. Obtaining 1000 examples doesn't necessarily mean a person has to type each character laboriously.

You could gather a custom dataset from other users for a fee or for free. It also could have been made more user-friendly by passively gathering data as the user goes about their regular typing experience, instead of mundanely having to type each letter x amount of times.

Read_from_serial reads the bytes until "\n". I calculated this to be ~30, but this particular area of the code could improve. What I figured was happening is that there are ten characters for each integer and character coming in from the gloves, thus amounting to 30 bytes.

Since characters are 1 byte, and the low byte is used on integers coming in from the Arduinos, making them 1 byte as well. It's then appended to the payload containing b''.

Then the object serprocobj is instantiated from the serprocedure class.

# read from serial port
def read_from_serial(serial,board,key,flag):
    #print("reading from {}: port {}".format(board, port))
    payload = b''
    
    #bytes_count = 0

	# CHANGED BYTES_TO_READ to inWaiting implementation
    #inwaitingbytes = serial.in_waiting()
	# Changed to  <= to make it count the last byte
    #while bytes_count <= inwaitingbytes:
        #read_bytes = serial.read(1)

		# sum number of bytes returned (not 2), you have set the timeout on serial port
		# see https://pythonhosted.org/pyserial/pyserial_api.html#serial.Serial.read
        #bytes_count = bytes_count + len(read_bytes)
    read_bytes = serial.read_until("\n",30)
    payload = payload + read_bytes

            # here you have the bytes, do your logic
            # Instantiate object from serprocedure class
    serprocobj = serprocedure(payload,flag,key)     
   
	# If the property THREE_TIMES_PROPERTY is greater than
    #print("READ from {}: [{}]".format(board,payload))
    return

def counter():
	if 'cnt' not in counter.__dict__:
		counter.cnt = 0
	else:
		counter.cnt += 1
	return counter.cnt

4. The Final Thread

The important ideas involved in main that may not immediately be obvious are the key fetching and threads. The key fetching is done by key = msvcrt.getch().

Remember when I told you earlier about certain things being blocking? The variables t, t1, and t.start(), t1.start() prevent the different serial calls from blocking, thus facilitating the normal progression of the code flow. reset_input_buffer is another important method.

I found out the hard way that without this, some of the data remained in the buffers after the process ran.

def main():
	while True:
		# For some reason it's only updating every other go
		# This will be the response of 1, 2, or 3
		if 'rsp' not in main.__dict__:
			# Ask what the user wants to train on 
			print("There are three options. Press 1 to train on alphabetic keys a-z, press 2 to train on space key, and press 3 to train on resting home-row position.")
			main.rsp = input("What would you like to do? ")
		# Depending on the answer received train on the given type
		if main.rsp == '1':
			if 'cnt' not in main.__dict__:
				main.cnt = 0
				# Request that the user type each letter 20 times for the machine learning dataset to train on
				main.ready = input("Please type each letter of the alphabet on the keyboard 20 times. Type y when ready and hit enter. ")
			elif main.ready == 'y':
				main.cnt += 1
				key = msvcrt.getch()
				if key and counter() <= 520:
					# Pass in the function, serial, board, and key as agrguments
					# We'll pass in a flag to identify which board is being used
					t = threading.Thread(target=read_from_serial, args=(ser0,"HC-06(Red)",key.decode('ASCII'),0))
					t1 = threading.Thread(target=read_from_serial, args=(ser1,"RN(Green)",key.decode('ASCII'),1))
					# Start the threads
					t.start()
					t1.start()
					# Be careful this is blocking. Gets the missing data amount
					#print(q.get())
					# wait for all threads termination
					# The joins may be holding up the buffer flushes, if they are move them to the bottom
					# Flush the serial input and output buffers
					ser0.reset_input_buffer()
					ser0.reset_output_buffer()
					ser1.reset_input_buffer()
					ser1.reset_output_buffer()
					t.join()
					t1.join()
				else:
					break
		elif main.rsp == '2':
			if 'cnt' not in main.__dict__:
				main.cnt = 0
				# Request permission to record homerow entries
				main.ready = input("Please place fingers in homerow positions and type space 20 times. Type j and hit enter. ")
			elif main.ready == 'j':
				main.cnt += 1
				# We don't need to record the key this time.
				if counter() <= 20:
					# Pass in the function, serial, board, and key as agrguments
					# We'll pass in a flag to identify which board is being used
					t = threading.Thread(target=read_from_serial, args=(ser0,"HC-06(Red)","homerow",0))
					t1 = threading.Thread(target=read_from_serial, args=(ser1,"RN(Green)","homerow",1))
					# Start the threads
					t.start()
					t1.start()
					# Be careful this is blocking. Gets the missing data amount
					#print(q.get())
					# wait for all threads termination
					# The joins may be holding up the buffer flushes, if they are move them to the bottom
					# Flush the serial input and output buffers
					ser0.reset_input_buffer()
					ser0.reset_output_buffer()
					ser1.reset_input_buffer()
					ser1.reset_output_buffer()
					t.join()
					t1.join()
				else:
					break
		elif main.rsp == '3':
			if 'cnt' not in main.__dict__:
				main.cnt = 0
				# Request the user record space 20 times
				main.ready = input("This step requires you to type the space key 20 times. To begin, press the y key and hit enter")
			elif main.ready == 'y':
				main.cnt += 1
				key = msvcrt.getch()
				# If ready equals y(yes) and counter is less than or equal to 20, else break
				if key and counter() <= 20:
					# Pass in the function, serial, board, and key as agrguments
					# We'll pass in a flag to identify which board is being used
					t = threading.Thread(target=read_from_serial, args=(ser0,"HC-06(Red)","space",0))
					t1 = threading.Thread(target=read_from_serial, args=(ser1,"RN(Green)","space",1))
					# Start the threads
					t.start()
					t1.start()
					# Be careful this is blocking. Gets the missing data amount
					#print(q.get())
					# wait for all threads termination
					# The joins may be holding up the buffer flushes, if they are, move them to the bottom
					# Flush the serial input and output buffers
					ser0.reset_input_buffer()
					ser0.reset_output_buffer()
					ser1.reset_input_buffer()
					ser1.reset_output_buffer()
					t.join()
					t1.join()
				else:
					break
	print("Thank you for training using this ML Gloves program")
	# Added serial close to the program
	ser0.close()
	ser1.close()

main()

Thank You!

Thank you so much for reading this tutorial! You may see the completed project at Brock's Gists. For further in-depth information on this topic, you may want to visit my second tutorial on how to program machine learning typing gloves with Arduino. It can be found here: Arduino and Machine Learning Programming for ML Gloves

This article is accurate and true to the best of the author’s knowledge. Content is for informational or entertainment purposes only and does not substitute for personal counsel or professional advice in business, financial, legal, or technical matters.

© 2021 Brock Lynch

Related Articles