submitted 2 years ago by lntl@lemmy.ml to c/python@lemmy.ml

4 comments fedilink hide all child comments

I have a large object that I want to save to the disk because it takes a minute to generate. The OOM reaper kills the process while pickle.dump ing the object.

It's a tuple of dicts of tuple of array.array.

Can pickle dump in chunks? If not, is there another technique I can use?

you are viewing a single comment's thread
view the rest of the comments

[-] alehc@lemmy.ml 1 points 2 years ago

You mean numpy arrays? I think the most efficient way to store them is via np.save. You could try creating a new directory and store all of your arrays there with clever file naming to retrieve the dictionary structure later.

Alternatively if you are up to trying to use pytorch you can convert the arrays to tensors and use torch.save to save the entire dictionary in one file. Installing pytorch just for this might be a bit overkill as it is a >1GB installation tho.

[-] lntl@lemmy.ml 0 points 2 years ago

It's a tuple of dicts of tuples of array.array, no numpy or torch :(

[-] alehc@lemmy.ml 1 points 2 years ago* (last edited 2 years ago)

So python standard library lists?

this post was submitted on 09 Jun 2023

0 points (50.0% liked)

Python

3244 readers

1 users here now

News and discussions about the programming language Python

founded 5 years ago

MODERATORS

MinutePhrase@lemmy.ml