Fix memory usage on export #28

Closed
opened 2020-09-11 08:54:50 +00:00 by yulqen · 3 comments
yulqen commented 2020-09-11 08:54:50 +00:00 (Migrated from github.com)

Ref https://github.com/yulqen/datamaps/issues/11

Use of a blank template in region of 3/4Mb leads to a RAM max-out at around 18 templates and subsequent machine lock.

Test files are in dft/bcompiler_engine_issue_28.zip.

Ref https://github.com/yulqen/datamaps/issues/11 Use of a blank template in region of 3/4Mb leads to a RAM max-out at around 18 templates and subsequent machine lock. Test files are in `dft/bcompiler_engine_issue_28.zip`.
banillie commented 2020-09-11 11:26:47 +00:00 (Migrated from github.com)

Thanks Matt.

Do you know which part of the template requires some much memory? I think in the past you have used a method for looking into workbooks to see which worksheets hold a lot of memory. If you can explain/show me how you do that, that would be very useful.

Thanks Matt. Do you know which part of the template requires some much memory? I think in the past you have used a method for looking into workbooks to see which worksheets hold a lot of memory. If you can explain/show me how you do that, that would be very useful.
yulqen commented 2020-09-11 12:51:42 +00:00 (Migrated from github.com)

"sheet16.xml" (which I think is 5A...) is 76Mb uncompressed for some reason. There were millions of empty rows in the XML file - to the point where I couldn't jump to the bottom of the file in my editor. However I was able to delete them and resave and I've just run the export and it ran the lot in about 3 minutes. I don't know what's caused this. I'll send everything over to you, including the fixed blank_template.xlsm. Be warned - that file may now be even more unstable due to me rippings it's internals out, but it seems to have worked for now.

"sheet16.xml" (which I think is 5A...) is 76Mb uncompressed for some reason. There were millions of empty rows in the XML file - to the point where I couldn't jump to the bottom of the file in my editor. However I was able to delete them and resave and I've just run the export and it ran the lot in about 3 minutes. I don't know what's caused this. I'll send everything over to you, including the fixed `blank_template.xlsm`. Be warned - that file may now be even more unstable due to me rippings it's internals out, but it seems to have worked for now.
banillie commented 2020-09-11 13:12:33 +00:00 (Migrated from github.com)

Thanks Matt. Files corrupting is an occupational hazard when working with excel.

Thanks Matt. Files corrupting is an occupational hazard when working with excel.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
yulqen/bcompiler-engine#28
No description provided.