Introducing Salesforce Einstein Vision: Einstein Vision, is a set of powerful new APIs that allow force.com and Heroku developers of all skill levels to bring image recognition to CRM and build AI-powered apps fast.
Wouldn't it be great to automatically identify keywords in customer reviews to help them determine problems. With the use of object detection companies can better understand customer preferences and lifestyle through their social media images. Also customers can take photos of in-store products to discover where they can make purchases online. This is possible with Salesforce Einstein Vision and its power of image recognition. Now everyone can leverage pre-trained classifiers such as food, scene, general or multi-label images in any app - with just a few clicks. We will take you through each step how to leverage pre-trained classifiers and get a prediction by Einstein.
The Einstein Vision
Einstein vision creates easy to build deep learning models for every use case including visual search, brand detection, and object identification with Einstein Image Classification. Salesforce Einstein Vision is a powerful machine learning APIs. Salesforce Einstein Vision is all about dataset, labels, training, models and prediction. In this blog post we will use Einstein Vision API and test how this API works to recognize the image and give predictions. We have gone through many Salesforce Einstein image prediction solutions. Now we try to use Einstein Vision API’s for the customer signature prediction.
This blog post takes you through each step of how to upload a customer signature image and create a prediction by Einstein. We will use Salesforce Einstein Vision API to upload signature image data sets in the Einstein Platform Service Account, train data sets and classify a signature image to get the Einstein prediction result.
We will design a model by creating a dataset, define the labels and provide signature images that belong to these labels. When we train the dataset, the system determines the differences between the various labels and generalizes the characteristics that define each label. When we send an image to the model, the model predicts which class that image falls into based on the classes specified in our training dataset.
Let's see how this works step by step with the Einstein Image Classification API and how to use it in Salesforce.
Step 1:
Einstein Platform Services Account Setup
First we need to setup an Einstein Platform Services Account. Sign Up for an Einstein Platform Services Account. Navigate to the sign-up page at https://api.einstein.ai/signup and click "Sign up using Salesforce".
When we sign up for an Einstein Platform Services account, the platform stores the email address from our Salesforce org. After successfull login, it will redirect to the Download Private Key Page. Click "Download Key"to save the key locally. The key file is named einstein_platform.pem. Now upload this key to Salesforce Files so that the Apex controller class can access it.
Step 2:
Einstein Platform Services APIs Connection
Generate an OAuth Access Token, the Einstein Platform Services APIs uses OAuth 2.0 JWT bearer token flow for authorization. This means that we need a valid token in each call we make to the API.
The API callout endpoint is: https://api.einstein.ai/v2/oauth2/token.
This API returns an OAuth access token and we must pass a valid access token in the header of each API call.
Request Example to generate an access token:
curl -H "Content-type: application/x-www-form-urlencoded" -X POST api.einstein.ai/v2/oauth2/token -d "grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer&assertion=<ASSERTION_STRING>"
Response Result from above request token api call:
"access_token":
"XLDZHXACW7DC45HQQLBZABCVFHWRT6KTBQNBIEAHCOYHCICM2Y34OBI46BD2K4XQQ2KLXI5HPHGT322G7MMKFOGE7534OGUMR6WC108",
"token_type": "Bearer",
"expires_in": 9999902
}
Step 3:
Create & train a dataset in Einstein Platform
Create and train the dataset to recognize and classify signature images using labels. We can create folders in a local system with labels and save the signature images inside of these folder. Now create a zip file of these folders. This zip is our dataset, we will use this dataset and upload it into Einstein Platform using apex API callout.
Create a dataset in Einstein Platform from a zip file. This API will create a dataset, labels, and images from the specified .zip file. The call returns immediately and continues to upload the images in the background.
The API callout endpoint is: https://api.einstein.ai/v2/vision/datasets/upload
Request API Example:
Create dataset from a local file
curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "data=@C:\Data\signs.zip" -F "type=image" https://api.einstein.ai/v2/vision/datasets/upload
Create dataset from a web file
curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "path=http://einstein.ai/images/signs.zip" -F "type=image" https://api.einstein.ai/v2/vision/datasets/upload
Response Result:
{
"id": 1000273,
"name": "Signs",
"createdAt": "2018-08-16T05:25:57.000+0000",
"updatedAt": "2018-08-16T05:25:57.000+0000",
"labelSummary": {
"labels": []
},
"totalExamples": 0,
"available": false,
"statusMsg": "UPLOADING",
"type": "image",
"object": "dataset"
}
Now use the Einstein Platform service API to train a dataset. This API trains a dataset and creates a model.
The API callout endpoint is: https://api.einstein.ai/v2/vision/train
Request Example:
curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "name=Signs" -F "datasetId=1000273" api.einstein.ai/v2/vision/train
Response Result:
"datasetId": 1000273,
"datasetVersionId": 0,
"name": "Beach and Mountain Model",
"status": "QUEUED",
"progress": 0,
"createdAt": "2018-08-16T05:25:57.000+0000”,
"updatedAt": "2018-08-16T05:25:57.000+0000",
"learningRate": 0.001,
"epochs": 3,
"queuePosition": 1,
"object": "training",
"modelId": "7JXCXTRXTMNLJCEF2DR5CJ46QU",
"trainParams": null,
"trainStats": null,
"modelType": "image"
}
Step 4:
Signature image prediction by Einstein Vision
Now for the moment of truth. We have created the dataset, trained the dataset to create a model, and at last are ready to test it.
Upload a signature image and use Einstein image prediction API. We can only get predictions using an image base64 string from models. The maximum image file size we can pass to this resource is 5 MB and the supported image file types are PNG, JPG, and JPEG. Predictions with the image file returns a prediction from an image model for the specified local image file.
The API callout endpoint:
Einstein Vision Predict Request Example:
curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "sampleId=Sign Prediction" -F "sampleContent=@/FileToPredict/badansign.jpg" -F "modelId=7JXCXTRXTMNLJCEF2DR5CJ46QU" api.einstein.ai/v2/vision/predict
Response Result:
{
"probabilities": [
{
"label": "Badan",
"probability": 0.76235858
},
{
"label": "Jon",
"probability": 0.06232360
},
{
"label": "Rick",
"probability": 0.07910490
},
{
"label": "Stony",
"probability": 0.09121290
}
],
"object": "predictresponse"
}
From the prediction response, we can see how the labels come into play when we make a prediction. Each probability value is always a number between 0 and 1. By default, the API returns all the labels in the model. For each label, the API returns the probability of an image belonging to that label.
For more information, please check out the Einstein Platform Services Developer Guide.