Prepare Your Data (Local Files or Bookmarks)

You need to decide on the source of data you’ll index (e.g., local files, PDFs, bookmarks).

Choose a Data Source

• Local files: Text files, PDFs, Word documents, etc.
• Bookmarks: You could use a browser extension to export your bookmarks in an easily parseable format (e.g., HTML, JSON).

Extract Text from Files

• For PDFs: You can use libraries like PyMuPDF or pdfminer to extract text from PDFs.
• For Word Documents: Use python-docx to extract text.
• For text files: Simply read the file content.

Example code to extract text from a PDF:

import fitz  # PyMuPDF

def extract_text_from_pdf(file_path):
    doc = fitz.open(file_path)
    text = ""
    for page in doc:
        text += page.get_text()
    return text

Index Data in OpenSearch

You can index documents by sending the extracted text to OpenSearch as JSON. Each document will be a JSON object with the relevant fields (e.g., title, content, date_created, etc.).

import os
import fitz  # PyMuPDF for PDFs
import docx  # python-docx for Word files
import requests
import json

# Function to extract text from PDF
def extract_text_from_pdf(file_path):
    doc = fitz.open(file_path)
    text = ""
    for page in doc:
        text += page.get_text()
    return text

# Function to extract text from Word files
def extract_text_from_docx(file_path):
    doc = docx.Document(file_path)
    text = ""
    for para in doc.paragraphs:
        text += para.text
    return text

# Function to extract text from plain text files
def extract_text_from_txt(file_path):
    with open(file_path, 'r', encoding='utf-8') as file:
        return file.read()

# Function to index document into OpenSearch
def index_document(file_path, content):
    url = "http://localhost:9200/personal-search/_doc/"
    doc = {
        "title": file_path,  # Or filename
        "content": content,  # Extracted text
        "date_created": "2025-04-01"  # Replace with actual creation date or timestamp
    }
    headers = {'Content-Type': 'application/json'}
    response = requests.post(url, headers=headers, data=json.dumps(doc))
    if response.status_code == 201:
        print(f"Successfully indexed: {file_path}")
    else:
        print(f"Error indexing: {file_path}")

# Function to process all files in a folder
def process_folder(folder_path):
    for root, dirs, files in os.walk(folder_path):
        for file in files:
            file_path = os.path.join(root, file)
            # Extract text based on file type
            if file.endswith(".pdf"):
                content = extract_text_from_pdf(file_path)
            elif file.endswith(".docx"):
                content = extract_text_from_docx(file_path)
            elif file.endswith(".txt"):
                content = extract_text_from_txt(file_path)
            else:
                print(f"Skipping unsupported file type: {file_path}")
                continue
            
            # Index the document in OpenSearch
            index_document(file_path, content)

# Example usage: pass the folder path
folder_path = "/path/to/your/folder"  # Replace with your folder path
process_folder(folder_path)

Search Functionality

OpenSearch provides a query DSL (Domain Specific Language) for searching your data. You can search for keywords, phrases, or use filters.

Example of a simple query:

def search_index(query):
    url = f"http://localhost:9200/personal-search/_search"
    search_query = {
        "query": {
            "match": {
                "content": query  # Searching within the 'content' field
            }
        }
    }
    response = requests.get(url, json=search_query)
    return response.json()

# Example of a search
result = search_index("Python tutorial")
print(result)

You can also use OpenSearch Dashboards as a UI to query the opensearch index.