Sunday, 15 September 2013

python read() and write() in large blocks / memory management

python read() and write() in large blocks / memory management

I'm writing some python code that splices together large files at various
points. I've done something similar in C where I allocated a 1MB char
array and used that as the read/write buffer. And it was very simple: read
1MB into the char array then write it out.
But with python I'm assuming it is different, each time I call read() with
size = 1M, it will allocate a 1M long character string. And hopefully when
the buffer goes out of scope it will we freed in the next gc pass.
Would python handle the allocation this way? If so, is the constant
allocation/deallocation cycle be computationally expensive?
Can I tell python to use the same block of memory just like in C? Or is
the python vm smart enough to do it itself?
I guess what I'm essentially aiming for is kinda like an implementation of
dd in python.

No comments:

Post a Comment