Skip to content
Snippets Groups Projects
Commit c9b0007f authored by FaustLyu's avatar FaustLyu
Browse files

remove whitespaces

parent a80fd111
No related branches found
No related tags found
No related merge requests found
......@@ -37,7 +37,7 @@ def process_jsonl_file(input_file, output_folder, chunk_size=100, filename='Unkn
for i, chunk in enumerate(chunks):
output_datum = {
'id': data['_id'] + '_' + str(i),
'text': chunk,
'text': chunk.strip(),
'title': ''
}
output_data.append(output_datum)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment