Vectorization & Embeddings Engineer

Job Description: Vectorization & Embeddings Engineer

Sidmans is seeking a skilled Vectorization & Embeddings Engineer with 4+ years of experience to join their team. This role focuses on the critical area of developing and optimizing robust pipelines for converting various data types into high-quality embeddings, a cornerstone for advanced AI and search functionalities.

Key Responsibilities

• Lead the design and development of scalable vectorization pipelines to convert text, PDFs, and multimodal data into high-quality embeddings using models like BGE, Ada, or Cohere.
• Manage and optimize vector databases such as Pinecone, Weaviate, Milvus, and Ravecter, including building efficient indexing strategies (e.g., HNSW, RiSKANN).
• Develop advanced chunking and parsing techniques to maintain context and improve retrieval accuracy within complex datasets.
• Build and fine-tune hybrid search workflows, combining the power of dense vector search with sparse keyword methods like BM25.
• Create comprehensive evaluation frameworks and gold-standard datasets to meticulously measure retrieval performance, focusing on metrics like Hit Rate, MRR, and Faithfulness.
• Implement sophisticated re-ranking pipelines utilizing cross-encoders to refine search results before they are delivered to Large Language Models (LLMs).
• Collaborate closely with engineering and product teams to ensure the development and deployment of scalable, high-performance retrieval systems.

Key skills/competency

• Vectorization
• Embeddings
• Pipeline Development
• Vector Databases (Pinecone, Weaviate)
• Indexing Strategies (HNSW, RiSKANN)
• Chunking & Parsing
• Hybrid Search
• BM25
• Cross-encoders
• LLMs Integration