PatentsView — Access to US Patents
In 2015, the USPTO set up a free Application Programming Interface (API) that allows programmatic access to over 7 million patents that were granted between 1976 to date [4]. You can use it to search for patents and it will return fields like patent number, grant date, title, abstract, inventor names, etc. Note that the database does not contain the full text of the patent nor any images.
Here is a sample call to retrieve the patent titles and abstracts for all patents granted since 2018.
https://api.patentsview.org/patents/query?q={"_gte":{"patent_date":"2018-01-01"}}&f=["patent_title","patent_abstract"]
Here is one of the real patents.
{patents:[...patent_title: "Audio assemblies for electronic devices", patent_abstract: "Voice-controlled devices that include one or more speakers for outputting audio. In some instances, the device includes at least one speaker within a cylindrical housing, with the speaker aimed or pointed away from a microphone coupled to the housing. For instance, if the microphone resides at or near the top of the cylindrical housing, then the speaker may point downwards along the longitudinal axis of the housing and away from the microphone. By pointing the speaker away from the microphone, the microphone will receive less sound from the speaker than if the speaker were pointed toward the microphone). Because the voice-controlled device may perform speech recognition on audio signals generated by the microphone, less sound from the speaker represented in the audio signal may result in more accurate speech recognition, and/or a lesser need to perform acoustic echo ccancelation(AEC) on the generated audio signals." ...]}
The patent text comes back in JavaScript Object Notation (JSON ) format. The results are returned in pages of 25 entries by default. You can change this and request up to 10,000 entries per page. More info about the API is here.
I used this API to grab the titles and abstracts of all 7+ million utility patents in the database. Next up I’ll show you how I preprocessed the text to train the NN.
KeyBERT — Keyword Extractor
The first step of preprocessing is to use an automated system to extract one to three keywords from the patent record that will summarize the type of invention. I will add these keywords to the training data so the NN will learn that keywords will be followed by the title and abstract of the patent. This will come in handy later. We will be able to generate new invention ideas when prompted by any set of keywords.
Patent titles often provide a short description of the invention. Sometimes the titles can be a bit wordy and contain legalese. So I used a variant of Google’s Bidirectional Encoder Representations from Transformers (BERT) [5] to extract keywords from the patent titles using an open-source package called KeyBERT [6]. Here’s a snippet of Python code.
Here are the results:
audio assemblies electronic
That’s a good, general summary for this type of patent.
Punkt — Sentence Tokenizer
The next step in preprocessing the text is getting a summary of the patent abstract. The abstracts can be a bit long. Grabbing the first two sentences is an easy way to summarize the text. I used the Punkt sentence tokenizer [7], part of the Natural Language Toolkit (NLTK), to break the abstract up into sentences.
Below is a sample of five preprocessed patents for training. Note that these are all real.
KEYWORDS: audio assemblies electronic
TITLE: Audio assemblies for electronic devices
ABSTRACT: Voice-controlled devices that include one or more speakers for outputting audio. In some instances, the device includes at least one speaker within a cylindrical housing, with the speaker aimed or pointed away from a microphone coupled to the housing.KEYWORDS: looped elastic leash
TITLE: Looped elastic leash
ABSTRACT: A looped elastic leash is a leash that is able to be stored around a neck of a pet animal, to be conveniently available to the user when walking or transporting animal. The looped elastic leash includes at least one elastic band, a covering fabric, a leash connector, and a collar-attachment hook.
KEYWORDS: oscillating fishing lure
TITLE: Oscillating fishing lure
ABSTRACT: A lure with an embedded spring is disclosed. The lure has an elastomeric body that is elongated.
KEYWORDS: double stranded rna
TITLE: Compositions and methods for the production and delivery of double stranded RNA
ABSTRACT: Compositions and methods for efficiently producing and delivering double stranded RNA (dsRNA) are provided. Vector constructs useful for in vitro and in vivo expression of dsRNA are described.
KEYWORDS: catcher helmet
TITLE: Catcher's helmet
ABSTRACT: A catcher's mask comprises a two-piece plastic shell composed of a rigid front shell and a rigid rear shell. The front and rear shells are connected by a strap harness attached to the front shell and have padding assemblies.
In the next session, I’ll show you how I trained the GPT-2 model to create new invention ideas.
In 2018 OpenAI created an NLP system called the Generative Pretrained Transformer (GPT) [8]. They later created an improved version of the transformer called GPT-2, which has sufficient capacity to infer meaning from text and perform common language tasks [9].
Training the Model
Once the patents are preprocessed using the techniques in the previous section, training GPT-2 is straightforward using Max Woolf’s GPT-2 Simple open-source project [10]. Here’s the code.
I used the large configuration of GPT-2 which has 744 million parameters. I trained it for 10,000 steps, taking about 8 hours on Google Colab Pro.
We can now use the trained model to generate new ideas by prompting the model with keywords. Note that GPT-2 provides a setting called “temperature” that can change the quality and diversity of the generated text. Higher temperatures (e.g. 100) will have greater linguistic variety where lower temperatures (e.g. 70) will be more grammatically correct.
For example, here’s the code to generate an idea for a household vacuum cleaner using a temperature of 85.
KEYWORDS: household vacuum cleaner
TITLE: Vacuum wall and floor cleaner
ABSTRACT: A vacuum cleaning system and apparatus contains a motor, an evaporating chamber, and a control circuit configured to remove dirt particles from the surface of a floor or a surface of a wall. The system is configured to control a supply of mild cleaning fluid in an evaporating chamber. The controller controls the supply of mild cleaning fluid in the evaporating chamber to an airstream and controls an air supply and a pressure in response to the airstream.
OK, not bad. It’s a bit wordy, but we can see that it is a vacuum cleaner for cleaning floors and walls using a mild cleaning fluid. Let’s see if we can clean up the abstract using a summarizing NN.
T5 Text Summarizer
I use a transformer from Google dubbed the “Text-to-Text Transfer Transformer”, known as T5 (get it?), to create more concise summarizations of the generated abstracts [11]. This model will create new text with a size limit to pare down the abstract. I’m using the Transformers module from Hugging Face [12] to simplify the use of T5. Here’s the Python code.
Here’s the resulting summary generated by T5-large that seems to capture the essence of the idea without all the repetition.
KEYWORDS: household vacuum cleaner
TITLE: Vacuum wall and floor cleaner
ABSTRACT: A vacuum cleaning system and apparatus contains a motor, an evaporating chamber, and a control circuit configured to remove dirt particles from the surface of a floor or a wall. The system is configured to control a supply of mild cleaning fluid to an airstream.
Here are four more results from InventorBot using “household vacuum cleaner” as the prompt and a temperature of 85. Note that I cherry-picked the best five out of 60 generated ideas.
KEYWORDS: household vacuum cleaner
TITLE: Multi-purpose unit for cleaning a wardrobe and the like
ABSTRACT: Garment-cleaning apparatus for use with clothing, particularly a scarf. Apparatus includes vacuum cleaner and a cleaning apparatus. Vacuum cleaner can be used for cleaning a shoulder of a wearer.KEYWORDS: household vacuum cleaner
KEYWORDS: household vacuum cleaner
TITLE: Compact apartment kitchen cabinet with high-efficiency household vacuum cleaner
ABSTRACT: Cabinet includes vertically extending side hose for the dust breathing cylinders, described herein is a means of housing a dust container having a lid, an accessory which can be used as a liner, and a bag or container.
TITLE: Household vacuum cleaner filter
ABSTRACT: Improved vacuum cleaner filter media from a number of different materials. Filter is cleaned by passing through the nozzle/piston assembly. Air diffuser conditions the filter media by means of an electric motor.KEYWORDS: household vacuum cleaner
TITLE: Improved residential vacuum cleaner
ABSTRACT: Indoor/outdoor vacuum cleaner includes a housing, a primary motor and an auxiliary motor. The dust container has a fan drive shaft operable to centrifugal force. Fan shaft test circuit is connected to the motor shaft and dust container.
Be sure to check out the appendix below for more examples of new ideas from InventorBot.
Quality of Results
The system seems to work fairly well. I find that about 5% of the generated ideas have new concepts and are well written, and about 10% of them at least have a germ of a novel idea. The other 90% are boring and/or poorly written.
Uniqueness
Because the training set of 7 million US patents is so large, the system does not seem to outright plagiarize any of the patents in the training set. I checked all of the examples in this article, and the generated text seems to be unique, although many of the concepts are not new.
Fair Warning
Note that the InventorBot system does not do an automated test for uniqueness, so if you generate any new ideas that seem actionable, I recommend that you run a patent search first.
Running the summarizer on the original abstracts before training would probably improve the results and may obviate the need to run the summarizer as a post-process. Also, using a larger NLP model like GPT-3 from OpenAI or the new Switch Transformer from Google may create better results.
Another way to possibly improve the invention ideas would be to include the patent dates in the training text and then prompt the generator with “DATE: 2021 KEYWORDS: …”. The system may learn the difference between old patents and new patents and then produce more modern ideas.
It may be possible to create an automatic scoring system that would rate each idea using two criteria: the quality of the text describing the invention and the uniqueness of the idea. This would make the cherry-picking easier.
You can run InventorBot on Google Colab. Note that you will need a Google account to run it.
- Click on the link, here.
- Log into your Google account, if you’re not logged in already.
- Click the first Run cell button (hover over the [ ] icon and click the play button). A warning will appear indicating that this notebook was not created by Google.
- Click Run anyway to initialize the system. It takes about 5 minutes to download the models and configure the NNs.
- Type in some keywords, specify the temperature, choose the number of ideas, and hit the second Run cell button to see what ideas InventorBot will generate.
- Repeat step 5 to generate more ideas.
All source code for this project is available on GitHub. You can experiment with the code using this Google Colab. The sources are released under the CC BY-SA license.
A special tip-o-the-cap to Oliver Strimpel who planted the seed that grew into the central idea for this project and provided feedback for the article. I would like to thank Jennifer Lim for her help with the project and for editing the article and Vahid Khorasani Ghassab and Mahsa Mesgaran for reviewing the final draft and providing feedback.
[1] Thaler, S., “Device Autonomously Bootstrapping Uniform Sensibility”, January 2015, https://patents.google.com/patent/US20150379394A1/en
[2] Thaler, S., “Devices and methods for attracting enhanced attention”, November 2017, https://patents.google.com/patent/EP3563896A1/en
[3] USPTO, “RE: Application No.: 16/524.350, Decision on Petition for Devices and Methods for Attracting Enhanced Attention”, April 2020, https://www.uspto.gov/sites/default/files/documents/16524350_22apr2020.pdf
[4] USPTO, PatentsView API, September 2015, https://api.patentsview.org/patent.html
[5] Devlin, J., Chang, M., Lee K., Toutanova, K., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, October 2018, https://arxiv.org/pdf/1810.04805.pdf
[6] Grootendorst, M., “KeyBERT”, https://github.com/MaartenGr/KeyBERT
[7] Bird, S., Loper, E., Nothman, J., Darcet, A., Punkt Sentence Tokenizer, 2012, https://www.nltk.org/_modules/nltk/tokenize/punkt.html
[8] Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I., “Improving Language Understanding by Generative Pre-Training”, June 2018
[9] Radford, A., Wu J., Child R., Luan, D., Amodei, D., Sutskever, I., “Language Models are Unsupervised Multitask Learners”, February 2019, https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
[10] Woolf, M., “GPT-2 Simple”, April 2019, https://github.com/minimaxir/gpt-2-simple
[11] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”, July 2020, https://arxiv.org/pdf/1910.10683.pdf
[12] Hugging Face, Transformers, https://huggingface.co/transformers/
Improved Headphones
Keywords: improved headphones
Temperature: 85
Cherry-picked: Best 5 from 20 results
KEYWORDS: improved headphones
TITLE: Improved headphones with facial mask and microphone
ABSTRACT: Various embodiments of devices and methods for improving headphones are described. In one embodiment a facial mask is provided with an attachement for a fitted first and second ear bud. In another embodiment, the apparatus may include at least one microphone.KEYWORDS: improved headphones
TITLE: System and method for improved headphones
ABSTRACT: A system for providing improved sound quality to a human ear is provided. The system includes an attachment for a drum, a sound attenuator, and an ear cover. Sound quality deterioration may be prevented by achieving insulated separation of the sound to be transmitted and received by the ears.KEYWORDS: improved headphones
TITLE: Headphone with improved controls
ABSTRACT: Headphone systems, methods, and computer program products for adjusting sound output according to a user's motion. Sound output can be adjusted in a gestural manner.KEYWORDS: improved headphones
TITLE: System for increased headphone bandwidth with a method for operating with wireless communication
ABSTRACT: Various embodiments of the present invention provide for the network of headphones used in wireless communication networks. In various embodiments, there is a system for providing direct audio signals to one or more headphones from a wireless network.KEYWORDS: improved headphones
TITLE: Method for improving headphones
ABSTRACT: Apparatus includes a sound source, two speakers for combining sound from sources of sound, a filter disposed in the sound source to adjust for a first resonant frequency, and a circuit for transferring sounds to the ears of a person. Apparatus includes an adjustment mechanism to account for the individual's resonant frequency.
Mobile App Driving Instructions
Keywords: mobile app driving instructions
Temperature: 80
Cherry-picked: Best 5 from 60 results
KEYWORDS: mobile app driving instructions
TITLE: Mobile app for driving instructions
ABSTRACT: A mobile app has a device with a camera and a controller. The controller is configured to receive sensor data indicative of an operational environment.KEYWORDS: mobile app driving instructions
TITLE: Mobile app for driving instructions with teaching guidance
ABSTRACT: Mobile apparatuses, methods, and computer-readable media are provided for assisting in driving instructions, applications and methods. The teaching guidance can be used to assist learner drivers in the mobile apparatus.KEYWORDS: mobile app driving instructions
TITLE: Mobile application for driving instructions
ABSTRACT: A method and a system for providing improved instructions to a user can include storing instructions for at least one vehicle. These instructions may be associated with the vehicle.KEYWORDS: mobile app driving instructions
TITLE: Managing a mobile device with system having two or more display devices
ABSTRACT: Mobile application may include a computing device having a first display and a second display.KEYWORDS: mobile app driving instructions
TITLE: Mobile app driving operations method, mobile app, and device of the mobile app
ABSTRACT: Application programming interface is configured to receive a driver behavior request from a mobile station. Mobile station communicates signals to and from a smartdevice.
Improved Cookware
Keywords: improved cookware
Temperature: 70
Cherry-picked: Best 5 from 40 results
KEYWORDS: improved cookware
TITLE: Improved cookware apparatus
ABSTRACT: Apparatus for cooking foodstuffs in a microwave oven includes a plate having a bottom surface defining an interior portion. A plurality of elongated ribs extend upwardly from the bottom surface.KEYWORDS: improved cookware
TITLE: Improved cookware
ABSTRACT: Skillet appliance of the present invention includes a cooking surface and a heat exchanger chamber. Container is adapted to utilize a heat exchange chamber to heat a food. Heating device for heating the food to be cooked in the container is also provided.KEYWORDS: improved cookware
TITLE: Method and apparatus for optimizing the cooking of the food products of an improved cookware pot
ABSTRACT: The present invention is directed to a method and apparatus for optimizing the cooking of a food product of an improved cookware pot to maximize the cooking efficiency. The cooking apparatus includes a heating device for heating the food product and a thermometer for detecting the temperature of the food.KEYWORDS: improved cookware
TITLE: Bread machine with improved cookware
ABSTRACT: Device and method for cooking bread in a bread machine is provided. The bread machine includes a rotary drum and a rotating member.KEYWORDS: improved cookware
TITLE: Method of and apparatus for reducing the heat loss in an improved cookware
ABSTRACT: A method of reducing heat loss in an improved cooking system is disclosed. The method comprises exposing a cooking surface of a pan to a heat loss reducing gas (w) in the cooking surface, and radiating the gas to the surface. A cavity of the pan is provided with a heating element to receive the gas. As the gas circulates and radiates, the gas reduces heat loss from the pan.