The general algorithm is this:
func importRecipe(image):
let allText = AzureCognitiveServices.ReadText(image);
let recipe = ChatGPT.CreateImportRecipeFunctionCall(allText);
database.LinkIngredients(recipe);
database.Save(recipe);
Let’s break that down even further!
Azure has a really good OCR API. You can try it out yourself here. Given an image, it will produce an object of all the pages, lines, and words within that image:
For our purposes, we simply take the text inside the image and concatenate it all together to form one big string that we will pass to ChatGPT. I found that replacing new-lines (\n) with spaces produced better results with ChatGPT, maybe the new-line tokens throw off the model.
Recently ChatGPT included a new function calling feature. Essentially it’s a more reliable way of transforming a prompt into JSON. You must supply ChatGPT with a JSONSchema definition of a function’s arguments (as an object), a prompt that asks ChatGPT to do something that may require calling that function, and then let ChatGPT do its magic. For CookTime’s case, I defined a function like this (this is TypeScript like but pretend it’s JSONSchema):
void ImportRecipe(recipe:
{
name: string,
servings: number,
cookTime: number,
steps: [
{
text: string
}
]
ingredientRequirements: [
{
ingredientName: string,
quantity: string,
unit: string
}
]
})
To note:
$refs
in your JSONSchema, they seem to be ignored.description
property of the JSONSchema does affect how ChatGPT interprets the text. For example, sometimes ingredient requirements would have a name like "tbsp. olive oil"
so I had to change the property description to say something like "ingredient name without units, or abbreviations"
. If you think AI is going to take everyone’s jobs, just remember even AI needs to be given precise instructions about what to do!Finally, the prompt I used to call ChatGPT went something like: Import the following as a recipe: [VISION API RESULTS CONCATENATED]
.
ChatGPT then returned an object according to my JSONSchema that was trivially convertible to the database recipe type.
The output of ChatGPT may or may not reference ingredients that CookTime already knows about. For example, the recipe shown in this post references olive oil. It’s important that the olive oil referenced by the recipe object is the existing olive oil ingredient in the database and not a duplicate. This merging/deduplication process already happens when a recipe is updated, but now we needed to run it even during creation.
An import does take about 30 seconds, but I need to deploy it to production and see how long it takes once everythign is in the cloud. I should come back here and update this with some numbers in the future.
This was the result of maybe 4-5 days of work, though I did already have experience working with the Vision API and ChatGPT from some work projects. Most of the time was spent refining the prompt and function schema for ChatGPT to produce the right results, and there may be an opportunity there to save time for more complex objects.
]]>Here are some examples:
Pretty cool!
]]>This post will be a development diary of all the changes necessary to accomplish this goal. Let’s dive in!
I’m going to use create-react-app to scaffold the initial state of the client app.
I did this by running npx create-react-app client-app --template typescript
at the root of the repo, which gets a React SPA set up in the client-app
directory.
Next I need to integrate the frontend development server with the backend server.
This is required only for development though setting it up is a little tedious and confusing so I’ll try my best to explain it to future me.
In production, the ASP.NET server would be serving the SPA as static pre-compiled HTML, JS, and CSS.
This is great because clients can cache resources to make page loads faster and server resources are mostly spent on serving REST calls instead of rendering HTML for the client.
However this means that if we want HMR, the ASP.NET server has to open and maintain a connection to browser clients and push changes to files as they are made.
This used to be the case with Microsoft.AspNetCore.SpaServices.Extensions
and to configure it you would have to add startup code like this:
app.Map(
"/js", ctx =>
{
ctx.UseSpa(spa =>
{
spa.UseProxyToSpaDevelopmentServer("http://localhost:8080/js");
});
});
This tells ASP.NET that any call to a path starting with /js
should be proxied to http://localhost:8080/js
where, hopefully, a second server is running listing for filesystem events to deliver the nice HMR experience.
In this case, ASP.NET is the first process that receives calls and decides to handle them or proxy them to another process.
The ASP.NET team decided they don’t want to maintain this anymore citing “simplicity” arguments so we have to now invert which process handles the request first:
/js
) the request is proxied to the React development server./api
) the request is proxied to the ASP.NET development server.Same result? Yes. Better? Probably not. Does it make the web development world harder by deprecating a working solution? Yes!
So we replace Microsoft.AspNetCore.SpaServices.Extensions
with Microsoft.AspNetCore.SpaProxy
and modify our csproj
file to tell it how to start the development server.
The full documentation for the new NuGet is here and that page does a better job than me.
With all that done, the standard create-react-app
page loads with the spinning Rutherfordian atom.
Success!
Before we get to the fun parts of re-writing the Razor pages as React components, we have to deal with the un-fun features of signing in and authorizing users. Addressing this led me down a rabbit-hole of authentication acronymns and open-ended questioning whether or not you should be able to use your Google account to sign in. In the end, I decided that CookTime will
The alternative of course would have been to
I simply do not have the patience or desire to setup OAuth authentication for CookTime and if this costs us users then so be it. The whole process is mind-meltingly boring.
Having decided that, I need to reimplement the Identity flows using React components instead of forms
for
If you use cookie authentication, the signin process just involves the server setting a cookie in a response.
Here’s what happened:
.AspNetCore.Identity.Application
cookie.Cookie
header.So how does the server authenticate the user and set the cookie? Something like this:
[HttpPost("signin")]
public async Task<IActionResult> SignIn(
[FromForm] SignIn signinRequest)
{
this.logger.LogInformation("Model state is valid, attempting login");
var user = signinRequest.UserNameOrEmail.Contains('@') ?
await this.userManager.FindUserByEmail(signinRequest.UserNameOrEmail) :
await this.userManager.FindUserByUserName(signinRequest.UserNameOrEmail);
// This is the line that sets the cookie in the response!!!
var result = await this.signInManager.SignInWithUserName(
userName: user.UserName,
password: signinRequest.Password,
isPersistent: signinRequest.RememberMe,
lockoutOnFailure: true);
if (result.Succeeded)
{
this.logger.LogInformation("User {Email} logged in", signinRequest.Email);
return this.Ok();
}
else
{
return this.BadRequest();
}
}
An interesting feature of this particular cookie is that it has the HttpOnly
flag set, which means that JavaScript code cannot access this cookie. Even if the client side code could access the cookie, it is encrypted by the server and it wouldn’t be feasible to decrypt it without leaking keys. Without access to a decrypted cookie, our client-side code does not have any actual information about the user. We don’t know their name, what roles they have, any other claims they may have, among others.
To solve this, we need an authenticated request to a route like /profile
that basically exchanges a valid authentication cookie with a JWT-like object that contains important claims like the username, roles, claims, etc… So that’s what we will do.
The registration form is “simple” (Ha! Nothing with authentication is simple!). A user providers the following information (taken from the current sign up form).
Then in the backend, we take this and check to see if a user with the same username
or email
has previously been registered.
Assuming not, we try to create the user and if their password passes the complexity checks then they get a database entry in the AspNetUsers
table.
We also fire off an email to the email provided to validate that this is a real email!
It should be simple enough to:
… A few days later … Indeed I did that and spent a few days in the refactoring jungles so now I will summarize some of the largest changes:
Now that we have a SPA, we have to be able to navigate between pages. The most popular library for React to do this seems to be Remix’s react-router so I’ve chosen to use it. The library works like this:
router
object that maps URLs to React component. Previously we relied on Razor page file organization conventions to match URL to Razor pages but now this is done explicitly by the router configuration.href
s with a combination of To
and Link
elements from react-router
. Normally, href
s make the browser navigate which is verboten for react-router. If you navigate, you are no longer a single-page application!Having understood react-router, the challenge here was actually rewriting every Razor page as a React page. The UI side of this was really easy, the problem was that we were running a lot of Entity Framework queries in the backend that were never exposed as REST APIs. For most of the pages I had to do something like this:
Simple work but it was very laborious.
Any recipe created in the last week will be shown in the New Recipes
list of the home page.
Usually these recipes are “in-development” so showing them first makes the author’s job easier.
For the featured recipes lists, we query our recipe database for recipes with photos, take N
of them, and then display them on the page.
It is preferable to ask the database to randomly sort and take the top N
recipes after sorting, otherwise we have to load all the recipes client-side, randomize, and take N
of them.
This can take an unnacceptable amount of time if the number of recipes is large.
CookTime uses PostgreSQL as its database, Entity Framework as the ORM, and the npsql as the bridge between LINQ queries and PostgreSQL statements.
PostgreSQL can certainl do what we want, I found a stack overflow question asking for exactly this: https://stackoverflow.com/questions/654906/linq-to-entities-random-order
The only missing piece was to enable an extension on the database to allow it to generate UUIDs on-demand to create random sort orders. The query-command is this:
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
With all that database infrastructure in place, we now show featured recipes on the home page :).
The home page and query results are now paginated. This should reduce the amount of scrolling needed on mobile devices to view recipes. We observed that users get lost if the page is too long to scroll through; pages should address this issue.
If you have a CookTime account, you can now rate and review recipes. Your ratings will be aggregated and averaged. Over time, we will come to find which recipes are the best recipes :)
You can now use your email OR your username to sign in to CookTime. We now also display validation errors during the sign up process. We have simplified the password requirements, basically you need atleast 2 unique characters and a minimum length of 6 total characters.
During user testing, we found that users wanted to know what were all of the categories of reicpes that CookTime had recipes for.
We did not have a way to view the list of categories, but now we do!
You can click on Categories
in the navigation bar and then click on a category you’re interested in to view all recieps for that category.
This feature is not user-facing but it did involve a big change in our backend so I’d like to describe it.
Early in CookTime’s development we identified a need for ingredient aliases. For example, “milk” and “whole milk” are practically the same ingredient, so for the purposes of auto-completion and building a correct recipe database we should treat them as the same ingredient. Initially this was not the case, recipes with “milk” and “whole milk” would be referencing unique ingredients and this would create several kinds of problems for us.
We could not associate different variants of an ingredient name with unique nutrition facts.
We could not properly highlight ingredient names in the recipe text if the user used one name in the ingredients list but another name in the text. This is extremely common: for example you might say a recipe need 6 galic cloves but then in the text of the recipe you will write “peel the garlic.” Since you did not write exactly “garlic clove”, we would not highlight the ingredient and thats not a good user experience.
Another problem was that we have a power user that creates recipes in portuguese so we had to associate the portuguese words for milk if we wanted nutrition facts and all the other features to work.
To implement aliases, we basically store a list of names for an ingredient along with a single “canonical” name.
In the case of milk, this looks like:
Milk; Leite; Whole Milk
Where Milk
is the canonical name, and Leite
(milk in portugues) and Whole Milk
are aliases for this ingredient.
The bulk of the work of this change was to hide from the user the fact that an ingredient’s name is now a semi-colon separated list of possible aliases.
]]>Favoriting is done by a clicking on a button in a recipe card.
You can view just your recipes by clicking on your name in the nav bar and then clicking on “My recipes”
]]>Now CookTime will automatically show you mass and volume for all ingredients. The conversion is very simple: if we know the density of the ingredient based on the USDA dataset, we will show you the opposite unit in SI unit. For example, 1 cup of milk will also be shown as 244g.
]]>Hi HN,
COVID lockdowns made me and my wife cook a lot more at home, and we had a need to streamline our recipe management woes. These were some of the problems we identified:
- Most recipe websites are what I call "mommy blogs", and the principal problem is that the recipe ingredient list and instructions are buried in a SEO-laden essay that we don't care about. We were sharing links to our favorite recipes, but links don't allow you to skip past the unnecessary essay prefacing the recipe. We knew we wanted a way to have recipes uncompromised by unrelated essays.
- Popular recipe aggregating websites are run by publishing companies that do not want you, the reader, to contribute your own recipes. This is not true for all of them; Allrecipes.com does allow you to enter recipes but that UI leaves a lot to be desired. We knew we needed a way for us (and you) to write your own recipes.
- Most recipe websites do not allow you to scale recipe ingredients. We like to cook a lot at once, and we don't like to keep track of a 2x, or 3x multiplier in our heads while cooking. We knew we wanted a simple way of scaling ingredients.
- Most recipe websites do not contain adequate nutrition information about their recipes. When they do, they don't show their math, or their sources. We knew we needed an automatic nutrition calculation for recipes based on ingredients and their quantities.
After a few months of nights and weekends, we are ready to share https://letscooktime.com . We solved all of the primary problems listed above, and more! Our features:
- Recipe creation: You can write your own recipes on CookTime, and share them with the world.
- Ingredient highlights: CookTime will highlight ingredients by name in the recipe's instructions, and if you click or tap on them we will show you the quantity needed. This becomes important when you're following a recipe on a phone because it saves you having to scroll back and forth between ingredient manifests and recipe text.
- Recipe components: Most recipes are simple enough to not need this feature, but some recipes are complex in a specific way: the same ingredient is used multiple times in different quantities. This seems to be more common in baking recipes; you may use flour or sugar in different components (like a pie crust vs. a filling) at different times with different quantities.
- Automatic nutrition calculation: CookTime contains the full dataset of the USDA Legacy Standard Reference nutrition dataset and part of the Branded Foods dataset. We associate ingredients with their corresponding entry in the dataset, scale based on mass, volume, or quantity, and then use that to compute nutrition facts for a recipe. Some ingredients may not have an association, and in that case we can fix that by editing the database directly (work in progress to improve this :))
- Grocery lists: You can add recipes to a grocery list, and if they have overlapping ingredients CookTime will simply add them up and show you a unified view of the groceries you need to buy.
- Scaling recipes: Nothing much to say here, it works.
- Tagging: The tagging system internally is generic enough, but we are currently limiting the set of allowed tags. Needless to say, you can tag recipes.
- No blogging, just recipes! No bullshit essay to scroll through.
We think the closest thing to CookTime is https://www.paprikaapp.com. Paprika is a paid native app, CookTime is a free mobile-ready website you can try _today_. I do not believe that recipe management really requires the performance, development cost, and App Store cost of a native app.
Without a CookTime account, you can:
- Browse recipes
- Read their nutrition facts
- Scale ingredients temporarily
- Share links to recipes
With a CookTime account, you can:
- Add your own recipes
- Add recipes to your personal groceries list
Let me know what you think! Would you use it? What are you currently using to track your recipes? What missing feature is stopping you from using CookTime?
And many more! To create an account, navigate to the Sign In page and fill in the form.
]]>