out of memory error when creating pptx file

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

out of memory error when creating pptx file

jaehoon jeong
Hello

I'm trying to generate a pptx file using the poi library.
The XMLSlideShow class contains about 600 XSLFSlides.
Each XSLFSlide object uses about 3mb of memory. 1.8gb of memory is required to create one pptx file.

It uses out too much memory, causing an out of memory error.
And I can't increase the heap memory size.

Is there a way to save slides in multiple times in one pptx file?
Is there a way to reduce memory usage?

Thanks in advance.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: out of memory error when creating pptx file

Rob Sargent
Since no one really wants to sit through 600 slides, break the total up in “chapters” and see where that gets you?

How much memory does you machine have, which version of java are you using.

> On Feb 3, 2020, at 7:15 AM, jaehoon jeong <[hidden email]> wrote:
>
> Hello
>
> I'm trying to generate a pptx file using the poi library.
> The XMLSlideShow class contains about 600 XSLFSlides.
> Each XSLFSlide object uses about 3mb of memory. 1.8gb of memory is required to create one pptx file.
>
> It uses out too much memory, causing an out of memory error.
> And I can't increase the heap memory size.
>
> Is there a way to save slides in multiple times in one pptx file?
> Is there a way to reduce memory usage?
>
> Thanks in advance.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: out of memory error when creating pptx file

kiwiwings
In reply to this post by jaehoon jeong
Hi,

XSLF and XmlBeans are undoubtedly memory hogs, but I can't solve this in a short period -
I often thought about an internal model which we could use for XSLF / HSLF but lets get back to your issue.

So when you say, you need to generate 600 slides ... does that mean they have similar structure?
i.e. could you use a template mechanism?

If true, I would generate one set of slides and then copy and fill those with the repetitive data outside of POI.
So handling the two XML files (slide + .refs) and adding it to the zip file is no magic.
Apart of the slides the presentation.xml and its .refs need also to be modified.

Andi



On 03.02.20 15:15, jaehoon jeong wrote:

> Hello
>
> I'm trying to generate a pptx file using the poi library.
> The XMLSlideShow class contains about 600 XSLFSlides.
> Each XSLFSlide object uses about 3mb of memory. 1.8gb of memory is required to create one pptx file.
>
> It uses out too much memory, causing an out of memory error.
> And I can't increase the heap memory size.
>
> Is there a way to save slides in multiple times in one pptx file?
> Is there a way to reduce memory usage?
>
> Thanks in advance.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: out of memory error when creating pptx file

jaehoon jeong
In reply to this post by Rob Sargent
Thank you for your reply.

I am using oracle jdk1.8. But I do not want to increase heap memory.
I want to find a solution in another way.

What does "chpaters" you mean?
For example, does it mean to divide into 100 slides and save each file?

I do not want to save as multiple files.
I want to save all slides in one file.

For example, I'm considering flushing slides to a file every 100 slides created.
Is this possible using the poi library?

On 2020/02/03 14:56:13, Rob Sargent <[hidden email]> wrote:

> Since no one really wants to sit through 600 slides, break the total up in “chapters” and see where that gets you?
>
> How much memory does you machine have, which version of java are you using.
>
> > On Feb 3, 2020, at 7:15 AM, jaehoon jeong <[hidden email]> wrote:
> >
> > Hello
> >
> > I'm trying to generate a pptx file using the poi library.
> > The XMLSlideShow class contains about 600 XSLFSlides.
> > Each XSLFSlide object uses about 3mb of memory. 1.8gb of memory is required to create one pptx file.
> >
> > It uses out too much memory, causing an out of memory error.
> > And I can't increase the heap memory size.
> >
> > Is there a way to save slides in multiple times in one pptx file?
> > Is there a way to reduce memory usage?
> >
> > Thanks in advance.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: out of memory error when creating pptx file

Rob Sargent


On 2/3/20 7:20 PM, jaehoon jeong wrote:
> Thank you for your reply.
>
> I am using oracle jdk1.8. But I do not want to increase heap memory.
> I want to find a solution in another way.
How little memory must this fit in?
>
> What does "chpaters" you mean?
> For example, does it mean to divide into 100 slides and save each file?
>
> I do not want to save as multiple files.
> I want to save all slides in one file.
>
> For example, I'm considering flushing slides to a file every 100 slides created.
> Is this possible using the poi library?
How far in to 600 do you get?
I'm not sure but to save each hundred slides would be to write a new
workbook.  You would then have to merge those into a single workbook via
poi unless you are prepared to merge the underlying xml files? That
approach would certainly be more memory efficient but more a xerces
problem than a poi problem.

>
> On 2020/02/03 14:56:13, Rob Sargent <[hidden email]> wrote:
>> Since no one really wants to sit through 600 slides, break the total up in “chapters” and see where that gets you?
>>
>> How much memory does you machine have, which version of java are you using.
>>
>>> On Feb 3, 2020, at 7:15 AM, jaehoon jeong <[hidden email]> wrote:
>>>
>>> Hello
>>>
>>> I'm trying to generate a pptx file using the poi library.
>>> The XMLSlideShow class contains about 600 XSLFSlides.
>>> Each XSLFSlide object uses about 3mb of memory. 1.8gb of memory is required to create one pptx file.
>>>
>>> It uses out too much memory, causing an out of memory error.
>>> And I can't increase the heap memory size.
>>>
>>> Is there a way to save slides in multiple times in one pptx file?
>>> Is there a way to reduce memory usage?
>>>
>>> Thanks in advance.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: out of memory error when creating pptx file

jaehoon jeong
Thank you for your reply.

I can use about 4g of memory. But I have to handle multiple requests.
For example, if three requests come in at the same time, an out of memory error occurs.

One request takes about 30 seconds.

Saving each of the 100 slides and then merging them also uses a lot of memory.

Can xerces merge individual pptx files without consuming too much memory?

On 2020/02/04 02:30:43, Rob Sargent <[hidden email]> wrote:

>
>
> On 2/3/20 7:20 PM, jaehoon jeong wrote:
> > Thank you for your reply.
> >
> > I am using oracle jdk1.8. But I do not want to increase heap memory.
> > I want to find a solution in another way.
> How little memory must this fit in?
> >
> > What does "chpaters" you mean?
> > For example, does it mean to divide into 100 slides and save each file?
> >
> > I do not want to save as multiple files.
> > I want to save all slides in one file.
> >
> > For example, I'm considering flushing slides to a file every 100 slides created.
> > Is this possible using the poi library?
> How far in to 600 do you get?
> I'm not sure but to save each hundred slides would be to write a new
> workbook.  You would then have to merge those into a single workbook via
> poi unless you are prepared to merge the underlying xml files? That
> approach would certainly be more memory efficient but more a xerces
> problem than a poi problem.
> >
> > On 2020/02/03 14:56:13, Rob Sargent <[hidden email]> wrote:
> >> Since no one really wants to sit through 600 slides, break the total up in “chapters” and see where that gets you?
> >>
> >> How much memory does you machine have, which version of java are you using.
> >>
> >>> On Feb 3, 2020, at 7:15 AM, jaehoon jeong <[hidden email]> wrote:
> >>>
> >>> Hello
> >>>
> >>> I'm trying to generate a pptx file using the poi library.
> >>> The XMLSlideShow class contains about 600 XSLFSlides.
> >>> Each XSLFSlide object uses about 3mb of memory. 1.8gb of memory is required to create one pptx file.
> >>>
> >>> It uses out too much memory, causing an out of memory error.
> >>> And I can't increase the heap memory size.
> >>>
> >>> Is there a way to save slides in multiple times in one pptx file?
> >>> Is there a way to reduce memory usage?
> >>>
> >>> Thanks in advance.
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: [hidden email]
> >>> For additional commands, e-mail: [hidden email]
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: out of memory error when creating pptx file

jaehoon jeong
In reply to this post by kiwiwings
Thank you for your reply.

The 600 slides have a similar structure.

Creating an xml file with template information and an xml file with data seems difficult.

But suppose I created each xml file.
How do I map the data?

I will parse the repetitive data xml and map it to a slides object. In the end, it will use a lot of memory.

Is there a way to make two xml files into one pptx file without using a lot of memory?

On 2020/02/03 19:21:51, Andreas Beeker <[hidden email]> wrote:

> Hi,
>
> XSLF and XmlBeans are undoubtedly memory hogs, but I can't solve this in a short period -
> I often thought about an internal model which we could use for XSLF / HSLF but lets get back to your issue.
>
> So when you say, you need to generate 600 slides ... does that mean they have similar structure?
> i.e. could you use a template mechanism?
>
> If true, I would generate one set of slides and then copy and fill those with the repetitive data outside of POI.
> So handling the two XML files (slide + .refs) and adding it to the zip file is no magic.
> Apart of the slides the presentation.xml and its .refs need also to be modified.
>
> Andi
>
>
>
> On 03.02.20 15:15, jaehoon jeong wrote:
> > Hello
> >
> > I'm trying to generate a pptx file using the poi library.
> > The XMLSlideShow class contains about 600 XSLFSlides.
> > Each XSLFSlide object uses about 3mb of memory. 1.8gb of memory is required to create one pptx file.
> >
> > It uses out too much memory, causing an out of memory error.
> > And I can't increase the heap memory size.
> >
> > Is there a way to save slides in multiple times in one pptx file?
> > Is there a way to reduce memory usage?
> >
> > Thanks in advance.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: out of memory error when creating pptx file

kiwiwings
With the current POI version, you'll end up with all slides in memory - I
think it could be possible to provide a lazy-loading interface where the
pages and shapes are parsed on the fly. But currently/as stated, I think the
solution is mostly without POI.

If you want you can send me an example (maybe just a few of the 600 slides)
privately and I'll have a look.

> Is there a way to make two xml files into one pptx file without using a
> lot of memory?
You need to do a few changes:
- renaming the slides
- adding the slides to presentation.xml + .refs
- in case you reference media files, update their filename and the slide
.refs




--
Sent from: http://apache-poi.1045710.n5.nabble.com/POI-User-f2280730.html

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: out of memory error when creating pptx file

Rob Sargent
In reply to this post by jaehoon jeong


> On Feb 4, 2020, at 1:32 AM, jaehoon jeong <[hidden email]> wrote:
>
> Thank you for your reply.
>
> I can use about 4g of memory. But I have to handle multiple requests.
Memory is remarkably inexpensive compared to your time and your sanity
> For example, if three requests come in at the same time, an out of memory error occurs.
>
> One request takes about 30 seconds.
>
Are you automatically generating 600slides per online request. Is the slideshow a report?
> Saving each of the 100 slides and then merging them also uses a lot of memory.
>
> Can xerces merge individual pptx files without consuming too much memory?

Xerces is an XML tool. Perhaps you can stream edit a template to generate the final show.

>> On 2020/02/04 02:30:43, Rob Sargent <[hidden email]> wrote:
>>
>>
>>> On 2/3/20 7:20 PM, jaehoon jeong wrote:
>>> Thank you for your reply.
>>>
>>> I am using oracle jdk1.8. But I do not want to increase heap memory.
>>> I want to find a solution in another way.
>> How little memory must this fit in?
>>>
>>> What does "chpaters" you mean?
>>> For example, does it mean to divide into 100 slides and save each file?
>>>
>>> I do not want to save as multiple files.
>>> I want to save all slides in one file.
>>>
>>> For example, I'm considering flushing slides to a file every 100 slides created.
>>> Is this possible using the poi library?
>> How far in to 600 do you get?
>> I'm not sure but to save each hundred slides would be to write a new
>> workbook.  You would then have to merge those into a single workbook via
>> poi unless you are prepared to merge the underlying xml files? That
>> approach would certainly be more memory efficient but more a xerces
>> problem than a poi problem.
>>>
>>> On 2020/02/03 14:56:13, Rob Sargent <[hidden email]> wrote:
>>>> Since no one really wants to sit through 600 slides, break the total up in “chapters” and see where that gets you?
>>>>
>>>> How much memory does you machine have, which version of java are you using.
>>>>
>>>>> On Feb 3, 2020, at 7:15 AM, jaehoon jeong <[hidden email]> wrote:
>>>>>
>>>>> Hello
>>>>>
>>>>> I'm trying to generate a pptx file using the poi library.
>>>>> The XMLSlideShow class contains about 600 XSLFSlides.
>>>>> Each XSLFSlide object uses about 3mb of memory. 1.8gb of memory is required to create one pptx file.
>>>>>
>>>>> It uses out too much memory, causing an out of memory error.
>>>>> And I can't increase the heap memory size.
>>>>>
>>>>> Is there a way to save slides in multiple times in one pptx file?
>>>>> Is there a way to reduce memory usage?
>>>>>
>>>>> Thanks in advance.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: [hidden email]
>>>>> For additional commands, e-mail: [hidden email]
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]