Generate image captions using Hugging Face
Use the Hugging Face Inference API to make calls to 100,000+ Machine Learning models from Supabase Edge Functions.
We can combine Hugging Face with Supabase Storage and Database Webhooks to automatically caption for any image we upload to a storage bucket.
About Hugging Face
Hugging Face is the collaboration platform for the machine learning community.
Huggingface.js provides a convenient way to make calls to 100,000+ Machine Learning models, making it easy to incorporate AI functionality into your Supabase Edge Functions.
Setup
- Open your Supabase project dashboard or create a new project.
- Create a new bucket called
images
. - Generate TypeScript types from remote Database.
- Create a new Database table called
image_caption
.- Create
id
column of typeuuid
which referencesstorage.objects.id
. - Create a
caption
column of typetext
.
- Create
- Regenerate TypeScript types to include new
image_caption
table. - Deploy the function to Supabase:
supabase functions deploy huggingface-image-captioning
. - Create the Database Webhook in the Supabase Dashboard to trigger the
huggingface-image-captioning
function anytime a record is added to thestorage.objects
table.
Generate TypeScript types
To generate the types.ts file for the storage and public schemas, run the following command in the terminal:
_10supabase gen types typescript --project-id=your-project-ref --schema=storage,public > supabase/functions/huggingface-image-captioning/types.ts
Code
Find the complete code on GitHub.
_49import { serve } from 'https://deno.land/std@0.168.0/http/server.ts'_49import { HfInference } from 'https://esm.sh/@huggingface/inference@2.3.2'_49import { createClient } from 'jsr:@supabase/supabase-js@2'_49import { Database } from './types.ts'_49_49console.log('Hello from `huggingface-image-captioning` function!')_49_49const hf = new HfInference(Deno.env.get('HUGGINGFACE_ACCESS_TOKEN'))_49_49type SoRecord = Database['storage']['Tables']['objects']['Row']_49interface WebhookPayload {_49 type: 'INSERT' | 'UPDATE' | 'DELETE'_49 table: string_49 record: SoRecord_49 schema: 'public'_49 old_record: null | SoRecord_49}_49_49serve(async (req) => {_49 const payload: WebhookPayload = await req.json()_49 const soRecord = payload.record_49 const supabaseAdminClient = createClient<Database>(_49 // Supabase API URL - env var exported by default when deployed._49 Deno.env.get('SUPABASE_URL') ?? '',_49 // Supabase API SERVICE ROLE KEY - env var exported by default when deployed._49 Deno.env.get('SUPABASE_SERVICE_ROLE_KEY') ?? ''_49 )_49_49 // Construct image url from storage_49 const { data, error } = await supabaseAdminClient.storage_49 .from(soRecord.bucket_id!)_49 .createSignedUrl(soRecord.path_tokens!.join('/'), 60)_49 if (error) throw error_49 const { signedUrl } = data_49_49 // Run image captioning with Huggingface_49 const imgDesc = await hf.imageToText({_49 data: await (await fetch(signedUrl)).blob(),_49 model: 'nlpconnect/vit-gpt2-image-captioning',_49 })_49_49 // Store image caption in Database table_49 await supabaseAdminClient_49 .from('image_caption')_49 .insert({ id: soRecord.id!, caption: imgDesc.generated_text })_49 .throwOnError()_49_49 return new Response('ok')_49})