OpenAI

What does it do?

The OpenAI – Embedding connector converts text into high-dimensional vector representations using OpenAI’s state-of-the-art embedding models. This node allows you to convert text into vector embeddings using OpenAI’s pre-trained models, capturing the semantic meaning of text and enabling similarity search, clustering, classification, and other advanced natural language processing tasks.

With the OpenAI – Embedding connector, you can:

• Transform text into vectors for use in vector databases or search engines
• Enable semantic search and content-based retrieval
• Prepare text data for downstream machine learning or AI workflows
• Cluster documents or sentences by meaning for organization or deduplication
• Feed text vectors into AI models for classification or anomaly detection
• Use embeddings in downstream tasks like similarity search or clustering

---

Inputs and Outputs

Inputs

• Text – Text content to convert to embeddings
• Documents – Document objects containing text to embed

Outputs

• Vectors – Generated vector embeddings
• Documents – Original documents with embeddings attached

---

How do I use it?

To use the OpenAI – Embedding connector in your workflow:

1. Add the OpenAI – Embedding Connector
   • Insert the node into your pipeline where you want to generate embeddings from text
2. Connect Input
   • Connect the input lane (text or documents) to your text source
   • This could be a file dropper, parser, chat input, or any text source
3. Configure Parameters
   • Configure your OpenAI API credentials and model settings
   • Adjust embedding model, batch size, and other options as needed
4. Connect Output
   • The connector outputs the generated text embeddings
   • Send these to downstream nodes for similarity search, clustering, or further analysis

---

Configuration

Parameter	Description	Options/Notes
Model	OpenAI embedding model	See model options table below
API Key (Token)	Enter your API key or token	Required for authentication

Available Models

UI Option	OpenAI Model Name	Description
Text Large	text-embedding-3-large	Powerful embedding model with highest accuracy and semantic understanding
Text Small	text-embedding-3-small	Highly efficient embedding model optimized for speed and performance
Text Ada	text-embedding-ada-002	Previous generation embedding model for backward compatibility with existing systems

---

Example Use Cases

• Enable semantic search or “find similar documents” features
• Cluster documents or sentences by meaning for organization or deduplication
• Feed text vectors into AI models for classification or anomaly detection
• Build recommendation systems based on content similarity
• Create knowledge bases with semantic search capabilities
• Perform content analysis and topic modeling
• Enable chatbots with context-aware responses
• Implement plagiarism detection or content similarity checks

---

Best Practices

Text Preparation

API Usage Optimization

---

API Considerations

---

Troubleshooting

API Problems

• Authentication errors – Verify API key validity
• Rate limit exceeded – Implement request throttling or upgrade API tier
• Timeout errors – Increase timeout setting or reduce batch size
• Network Errors – Verify internet connectivity and firewall settings
• Quota Exceeded – Check your OpenAI account usage and billing status

Embedding Quality Issues

• Poor semantic matching – Try a higher-dimensional model
• Inconsistent results – Standardize text preprocessing
• High latency – Optimize batch size or implement caching
• Text Length Issues – Ensure input text doesn’t exceed OpenAI’s token limits
• Model Errors – Verify the selected model is available and supported

---

Technical Reference

For detailed technical information, refer to:

• OpenAI Embeddings API Documentation
• OpenAI Embedding Models

---