Leveraging GPT-3.5 and unstructured APIs for translations<\/em><\/h4>\nThis blog post details how I utilised GPT to translate the personal memoir of a family friend, making it accessible to a broader audience.<\/strong> Specifically, I employed GPT-3.5 for translation and Unstructured\u2019s APIs for efficient content extraction and formatting.<\/p>\nThe memoir, a heartfelt account by my family friend Carmen Rosa, chronicles her upbringing in Bolivia and her romantic journey in Paris with an Iranian man during the vibrant 1970s. Originally written in Spanish, we aimed to preserve the essence of her narrative while expanding its reach to English-speaking readers through the application of LLM technologies.<\/p>\n![\"Cover](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2ATDfCHL6K8wYGU9fGiWHnWw.png?ssl=1\")
Cover image of \u201cUn Destino Sorprendente\u201d, used with permission of author Carmen Rosa Wichtendahl.<\/figcaption><\/figure>\nBelow you can read the translation process in more detail or you can access here the Colab Notebook.<\/a><\/p>\nTranslating the\u00a0document<\/h3>\n
I followed the next steps for the translation of the\u00a0book:<\/p>\n
\n- \nImport Book Data:<\/strong> I imported the book from a Docx document using the Unstructured API and divided it into chapters and paragraphs.<\/li>\n
- \nTranslation Technique: <\/strong>I translated each chapter using GPT-3.5. For each paragraph, I provided the latest three translated sentences (if available) from the same chapter. This approach served two purposes:<\/li>\n<\/ol>\n
\n- \nStyle Consistency:<\/em><\/strong> Maintaining a consistent style throughout the translation by providing context from previous translations.<\/li>\n
- \nToken Limit:<\/em><\/strong> <\/em>Limiting the number of tokens processed at once to avoid exceeding the model\u2019s context\u00a0limit.<\/li>\n<\/ul>\n
3. Exporting translation as Docx: <\/strong>I used Unstructured\u2019s API once again to save the translated content in Docx\u00a0format.<\/p>\nTechnical implementation<\/h3>\n1. Libraries<\/h4>\n
We\u2019 ll start with the installation and import of the necessary libraries.<\/p>\n
pip install --upgrade openai
pip install python-dotenv
pip install unstructured
pip install python-docx<\/pre>\nimport openai
# Unstructured
from unstructured.partition.docx import partition_docx
from unstructured.cleaners.core import group_broken_paragraphs
# Data and other libraries
import pandas as pd
import re
from typing import List, Dict
import os
from dotenv import load_dotenv<\/pre>\n2. Connecting to OpenAI\u2019s\u00a0API<\/h4>\n
The code below sets up the OpenAI API key for use in a Python project. You need to save your API key in an\u00a0.env\u00a0file.<\/p>\n
import openai
# Specify the path to the .env file
dotenv_path = '\/content\/.env'
_ = load_dotenv(dotenv_path) # read local .env file
openai.api_key = os.environ['OPENAI_API_KEY']<\/pre>\n3. Loading the\u00a0book<\/h4>\n
The code allows us to import the book in Docx format and divide it into individual paragraphs.<\/p>\n
elements = partition_docx(
filename=\"\/content\/libro.docx\",
paragraph_grouper=group_broken_paragraphs
)<\/pre>\nThe code below returns the paragraph in the 10th index of elements.<\/p>\n
print(elements[10])
# Returns: Destino sorprendente, es el t\u00edtulo que la autora le puso ...<\/pre>\n4. Group book into titles and\u00a0chapters<\/h4>\n
The next step involves creating a list of chapters. Each chapter will be represented as a dictionary containing a title and a list of paragraphs. This structure simplifies the process of translating each chapter and paragraph individually. Here\u2019s an example of this\u00a0format:<\/p>\n
[
{\"title\": title 1, \"content\": [paragraph 1, paragraph 2, ..., paragraph n]},
{\"title\": title 2, \"content\": [paragraph 1, paragraph 2, ..., paragraph n]},
...
{\"title\": title n, \"content\": [paragraph 1, paragraph 2, ..., paragraph n]},
]<\/pre>\nTo achieve this, we\u2019ll create a function called group_by_chapter. Here are the key steps involved:<\/p>\n
\n- \nExtract Relevant Information: <\/strong>We can get each narrative text and title by calling element.category. <\/em>Those are the only categories we\u2019re interested in translating at this\u00a0point.<\/li>\n
- \nIdentify Narrative Titles:<\/strong> We recognise that some titles should be part of the narrative text. To account for this, we assume that italicised titles belong to the narrative paragraph.<\/li>\n<\/ol>\n
def group_by_chapter(elements: List) -> List[Dict]:
chapters = []
current_title = None
for element in elements:
text_style = element.metadata.emphasized_text_tags # checks if it is 'b' or 'i' and returns list
unique_text_style = list(set(text_style)) if text_style is not None else None
# we consider an element a title if it is a title category and the style is bold
is_title = (element.category == \"Title\") & (unique_text_style == ['b'])
# we consider an element a narrative content if it is a narrative text category or
# if it is a title category, but it is italic or italic and bold
is_narrative = (element.category == \"NarrativeText\") | (
((element.category == \"Title\") & (unique_text_style is None)) |
((element.category == \"Title\") & (unique_text_style == ['i'])) |
((element.category == \"Title\") & (unique_text_style == ['b', 'i']))
)
# for new titles
if is_title:
print(f\"Adding title {element.text}\")
# Add previous chapter when a new one comes in, unless current title is None
if current_title is not None:
chapters.append(current_chapter)
current_title = element.text
current_chapter = {\"title\": current_title, \"content\": []}
elif is_narrative:
print(f\"Adding Narrative {element.text}\")
current_chapter[\"content\"].append(element.text)
else:
print(f'### No need to convert. Element type: {element.category}')
return chapters<\/pre>\nIn the example below, we can see an\u00a0example:<\/p>\n
book_chapters[2]
# Returns
{'title': 'Proemio',
'content': [
'La autobiograf\u00eda es considerada ...',
'Dentro de las artes literarias, ...',
'Se encuentra m\u00e1s pr\u00f3xima a los, ...',
]
}<\/pre>\n5. Book translation<\/h4>\n
To translate the book, we follow these\u00a0steps:<\/p>\n
\n- \nTranslate Chapter Titles:<\/strong> We translate the title of each\u00a0chapter.<\/li>\n
- \nTranslate Paragraphs:<\/strong> We translate each paragraph, providing the model with the latest three translated sentences as\u00a0context.<\/li>\n
- \nSave Translations:<\/strong> We save both the translated titles and\u00a0content.<\/li>\n<\/ol>\n
The function below automates this\u00a0process.<\/p>\n
def translate_book(book_chapters: List[Dict]) -> Dict:
translated_book = []
for chapter in book_chapters:
print(f\"Translating following chapter: {chapter['title']}.\")
translated_title = translate_title(chapter['title'])
translated_chapter_content = translate_chapter(chapter['content'])
translated_book.append({
\"title\": translated_title,
\"content\": translated_chapter_content
})
return translated_book<\/pre>\nFor the title, we ask GPT a simple translation as\u00a0follows:<\/p>\n
def translate_title(title: str) -> str:
response = client.chat.completions.create(
model=\"gpt-3.5-turbo\",
messages= [{
\"role\": \"system\",
\"content\": f\"Translate the following book title into English:n{title}\"
}]
)
return response.choices[0].message.content<\/pre>\nTo translate a single chapter, we provide the model with the corresponding paragraphs. We instruct the model as\u00a0follows:<\/p>\n
\n- \nIdentify the role:<\/strong> We inform the model that it is a helpful translator for a\u00a0book.<\/li>\n
- \nProvide context:<\/strong> We share the latest three translated sentences from the\u00a0chapter.<\/li>\n
- \nRequest translation:<\/strong> We ask the model to translate the next paragraph.<\/li>\n<\/ol>\n
During this process, the function combines all translated paragraphs into a single\u00a0string.<\/p>\n
# Function to translate a chapter using OpenAI API
def translate_chapter(chapter_paragraphs: List[str]) -> str:
translated_content = \"\"
for i, paragraph in enumerate(chapter_paragraphs):
print(f\"Translating paragraph {i + 1} out of {len(chapter_paragraphs)}\")
# Builds the message dynamically based on whether there is previous translated content
messages = [{
\"role\": \"system\",
\"content\": \"You are a helpful translator for a book.\"
}]
if translated_content:
latest_content = get_last_three_sentences(translated_content)
messages.append(
{
\"role\": \"system\",
\"content\": f\"This is the latest text from the book that you've translated from Spanish into English:n{latest_content}\"
}
)
# Adds the user message for the current paragraph
messages.append(
{
\"role\": \"user\",
\"content\": f\"Translate the following text from the book into English:n{paragraph}\"
}
)
# Calls the API
response = client.chat.completions.create(
model=\"gpt-3.5-turbo\",
messages=messages
)
# Extracts the translated content and appends it
paragraph_translation = response.choices[0].message.content
translated_content += paragraph_translation + 'nn'
return translated_content<\/pre>\nFinally, below we can see the supporting function to get the latest three sentences.<\/p>\n
def get_last_three_sentences(paragraph: str) -> str:
# Use regex to split the text into sentences
sentences = re.split(r'(?<!w.w.)(?<![A-Z][a-z].)(?<=.|?)s', paragraph)
# Get the last three sentences (or fewer if the paragraph has less than 3 sentences)
last_three = sentences[-3:]
# Join the sentences into a single string
return ' '.join(last_three)<\/pre>\n6. Book\u00a0export<\/h4>\n
Finally, we pass the dictionary of chapters to a function that adds each title as a heading and each content as a paragraph. After each paragraph, a page break is added to separate the chapters. The resulting document is then saved locally as a Docx\u00a0file.<\/p>\n
from docx import Document
def create_docx_from_chapters(chapters: Dict, output_filename: str) -> None:
doc = Document()
for chapter in chapters:
# Add chapter title as Heading 1
doc.add_heading(chapter['title'], level=1)
# Add chapter content as normal text
doc.add_paragraph(chapter['content'])
# Add a page break after each chapter
doc.add_page_break()
# Save the document
doc.save(output_filename)<\/pre>\nLimitations<\/h3>\n
While using GPT and APIs for translation is fast and efficient, there are key limitations compared to human translation:<\/p>\n
\n- \nPronoun and Reference Errors: <\/strong>GPT did misinterpret pronouns or references in few cases, potentially attributing actions or statements to the wrong person in the narrative. A human translator can better resolve such ambiguities.<\/li>\n
- \nCultural Context:<\/strong> GPT missed subtle cultural references and idioms that a human translator could interpret more accurately. In this case, several slang terms unique to Santa Cruz, Bolivia, were retained in the original language without additional context or explanation.<\/li>\n<\/ul>\n
Combining AI with human review can balance speed and quality, ensuring translations are both accurate and authentic.<\/p>\n
Conclusion<\/h3>\n
This project demonstrates an approach to translating a book using a combination of GPT-3 and Unstructured APIs. By automating the translation process, we significantly reduced the manual effort required. While the initial translation output may require some minor human revisions to refine the nuances and ensure the highest quality, this approach serves as a strong foundation for efficient and effective book translation<\/p>\n
If you have any feedback or suggestions on how to improve this process or the quality of the translations, please feel free to share them in the comments\u00a0below.<\/p>\n
Appendix<\/h3>\n\n- Link to Colab\u00a0Notebook<\/a>\n<\/li>\n
- Link to book in original language (Spanish)<\/a>\n<\/li>\n
- Link to translated book (English)<\/a>\n<\/li>\n<\/ul>\n
![\"\"](\"https:\/\/medium.com\/_\/stat?event=post.clientViewed&referrerSource=full_rss&postId=08913ca60020\")
<\/p>\n
\nTranslating a Memoir: A Technical Journey<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\n
The memoir, a heartfelt account by my family friend Carmen Rosa, chronicles her upbringing in Bolivia and her romantic journey in Paris with an Iranian man during the vibrant 1970s. Originally written in Spanish, we aimed to preserve the essence of her narrative while expanding its reach to English-speaking readers through the application of LLM technologies.<\/p>\n Below you can read the translation process in more detail or you can access here the Colab Notebook.<\/a><\/p>\n I followed the next steps for the translation of the\u00a0book:<\/p>\n 3. Exporting translation as Docx: <\/strong>I used Unstructured\u2019s API once again to save the translated content in Docx\u00a0format.<\/p>\n We\u2019 ll start with the installation and import of the necessary libraries.<\/p>\n The code below sets up the OpenAI API key for use in a Python project. You need to save your API key in an\u00a0.env\u00a0file.<\/p>\n The code allows us to import the book in Docx format and divide it into individual paragraphs.<\/p>\n The code below returns the paragraph in the 10th index of elements.<\/p>\n The next step involves creating a list of chapters. Each chapter will be represented as a dictionary containing a title and a list of paragraphs. This structure simplifies the process of translating each chapter and paragraph individually. Here\u2019s an example of this\u00a0format:<\/p>\n To achieve this, we\u2019ll create a function called group_by_chapter. Here are the key steps involved:<\/p>\n In the example below, we can see an\u00a0example:<\/p>\n To translate the book, we follow these\u00a0steps:<\/p>\n The function below automates this\u00a0process.<\/p>\n For the title, we ask GPT a simple translation as\u00a0follows:<\/p>\n To translate a single chapter, we provide the model with the corresponding paragraphs. We instruct the model as\u00a0follows:<\/p>\n During this process, the function combines all translated paragraphs into a single\u00a0string.<\/p>\n Finally, below we can see the supporting function to get the latest three sentences.<\/p>\n Finally, we pass the dictionary of chapters to a function that adds each title as a heading and each content as a paragraph. After each paragraph, a page break is added to separate the chapters. The resulting document is then saved locally as a Docx\u00a0file.<\/p>\n While using GPT and APIs for translation is fast and efficient, there are key limitations compared to human translation:<\/p>\n Combining AI with human review can balance speed and quality, ensuring translations are both accurate and authentic.<\/p>\n This project demonstrates an approach to translating a book using a combination of GPT-3 and Unstructured APIs. By automating the translation process, we significantly reduced the manual effort required. While the initial translation output may require some minor human revisions to refine the nuances and ensure the highest quality, this approach serves as a strong foundation for efficient and effective book translation<\/p>\n If you have any feedback or suggestions on how to improve this process or the quality of the translations, please feel free to share them in the comments\u00a0below.<\/p>\n Translating a Memoir: A Technical Journey<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\nTranslating the\u00a0document<\/h3>\n
\n
\n
Technical implementation<\/h3>\n
1. Libraries<\/h4>\n
pip install --upgrade openai
pip install python-dotenv
pip install unstructured
pip install python-docx<\/pre>\nimport openai
# Unstructured
from unstructured.partition.docx import partition_docx
from unstructured.cleaners.core import group_broken_paragraphs
# Data and other libraries
import pandas as pd
import re
from typing import List, Dict
import os
from dotenv import load_dotenv<\/pre>\n2. Connecting to OpenAI\u2019s\u00a0API<\/h4>\n
import openai
# Specify the path to the .env file
dotenv_path = '\/content\/.env'
_ = load_dotenv(dotenv_path) # read local .env file
openai.api_key = os.environ['OPENAI_API_KEY']<\/pre>\n3. Loading the\u00a0book<\/h4>\n
elements = partition_docx(
filename=\"\/content\/libro.docx\",
paragraph_grouper=group_broken_paragraphs
)<\/pre>\nprint(elements[10])
# Returns: Destino sorprendente, es el t\u00edtulo que la autora le puso ...<\/pre>\n4. Group book into titles and\u00a0chapters<\/h4>\n
[
{\"title\": title 1, \"content\": [paragraph 1, paragraph 2, ..., paragraph n]},
{\"title\": title 2, \"content\": [paragraph 1, paragraph 2, ..., paragraph n]},
...
{\"title\": title n, \"content\": [paragraph 1, paragraph 2, ..., paragraph n]},
]<\/pre>\n\n
def group_by_chapter(elements: List) -> List[Dict]:
chapters = []
current_title = None
for element in elements:
text_style = element.metadata.emphasized_text_tags # checks if it is 'b' or 'i' and returns list
unique_text_style = list(set(text_style)) if text_style is not None else None
# we consider an element a title if it is a title category and the style is bold
is_title = (element.category == \"Title\") & (unique_text_style == ['b'])
# we consider an element a narrative content if it is a narrative text category or
# if it is a title category, but it is italic or italic and bold
is_narrative = (element.category == \"NarrativeText\") | (
((element.category == \"Title\") & (unique_text_style is None)) |
((element.category == \"Title\") & (unique_text_style == ['i'])) |
((element.category == \"Title\") & (unique_text_style == ['b', 'i']))
)
# for new titles
if is_title:
print(f\"Adding title {element.text}\")
# Add previous chapter when a new one comes in, unless current title is None
if current_title is not None:
chapters.append(current_chapter)
current_title = element.text
current_chapter = {\"title\": current_title, \"content\": []}
elif is_narrative:
print(f\"Adding Narrative {element.text}\")
current_chapter[\"content\"].append(element.text)
else:
print(f'### No need to convert. Element type: {element.category}')
return chapters<\/pre>\nbook_chapters[2]
# Returns
{'title': 'Proemio',
'content': [
'La autobiograf\u00eda es considerada ...',
'Dentro de las artes literarias, ...',
'Se encuentra m\u00e1s pr\u00f3xima a los, ...',
]
}<\/pre>\n5. Book translation<\/h4>\n
\n
def translate_book(book_chapters: List[Dict]) -> Dict:
translated_book = []
for chapter in book_chapters:
print(f\"Translating following chapter: {chapter['title']}.\")
translated_title = translate_title(chapter['title'])
translated_chapter_content = translate_chapter(chapter['content'])
translated_book.append({
\"title\": translated_title,
\"content\": translated_chapter_content
})
return translated_book<\/pre>\ndef translate_title(title: str) -> str:
response = client.chat.completions.create(
model=\"gpt-3.5-turbo\",
messages= [{
\"role\": \"system\",
\"content\": f\"Translate the following book title into English:n{title}\"
}]
)
return response.choices[0].message.content<\/pre>\n\n
# Function to translate a chapter using OpenAI API
def translate_chapter(chapter_paragraphs: List[str]) -> str:
translated_content = \"\"
for i, paragraph in enumerate(chapter_paragraphs):
print(f\"Translating paragraph {i + 1} out of {len(chapter_paragraphs)}\")
# Builds the message dynamically based on whether there is previous translated content
messages = [{
\"role\": \"system\",
\"content\": \"You are a helpful translator for a book.\"
}]
if translated_content:
latest_content = get_last_three_sentences(translated_content)
messages.append(
{
\"role\": \"system\",
\"content\": f\"This is the latest text from the book that you've translated from Spanish into English:n{latest_content}\"
}
)
# Adds the user message for the current paragraph
messages.append(
{
\"role\": \"user\",
\"content\": f\"Translate the following text from the book into English:n{paragraph}\"
}
)
# Calls the API
response = client.chat.completions.create(
model=\"gpt-3.5-turbo\",
messages=messages
)
# Extracts the translated content and appends it
paragraph_translation = response.choices[0].message.content
translated_content += paragraph_translation + 'nn'
return translated_content<\/pre>\ndef get_last_three_sentences(paragraph: str) -> str:
# Use regex to split the text into sentences
sentences = re.split(r'(?<!w.w.)(?<![A-Z][a-z].)(?<=.|?)s', paragraph)
# Get the last three sentences (or fewer if the paragraph has less than 3 sentences)
last_three = sentences[-3:]
# Join the sentences into a single string
return ' '.join(last_three)<\/pre>\n6. Book\u00a0export<\/h4>\n
from docx import Document
def create_docx_from_chapters(chapters: Dict, output_filename: str) -> None:
doc = Document()
for chapter in chapters:
# Add chapter title as Heading 1
doc.add_heading(chapter['title'], level=1)
# Add chapter content as normal text
doc.add_paragraph(chapter['content'])
# Add a page break after each chapter
doc.add_page_break()
# Save the document
doc.save(output_filename)<\/pre>\nLimitations<\/h3>\n
\n
Conclusion<\/h3>\n
Appendix<\/h3>\n
\n
<\/p>\n
\n