POI memory usage degradation

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

POI memory usage degradation

Kirill Rajbhandary

Dear POI Users Group,

 

Unfortunately, I found out that the latest POI library poi-ooxml with versions 4.0.1, 4.1.0 consumes much more memory than the version 3.17 when it is required to write information to already existing large Excel file.

It requires approximately 6GB Java Heap Space to add these rows using poi-ooxml library with version 3.17 but it is not enough even twice more (12GB) with version 4.0.1 or 4.1.0. Besides, the performance is much worse with the latest versions of poi-ooxml. OpenJDK 11.0.02 was used to run this example.

In attachment the corresponding example with ExcelWriter class can be found that tries to add only 5 rows to already existing sheet “Sheet2” having only header row with 16 columns. The first sheet “Sheet1” of this file already contains 240K rows with 23 columns. Zipped ExcelWriter  with Test-Template.xslx can be downloaded from the next link:

https://drive.google.com/file/d/1vq6WvMv5mSFUzyJinN9aoYRBStxZuJCq/view

To run ExcelWriter main method, please specify the full path to Test-Template.xlsx and result file path. To change poi-ooxml version, it requires to change it in pom.xml, rebuild the project and run it again.

 

Could you please advise why it became worse comparing with the old library version poi-ooxml-3.17.jar and how it could be improved?

 

Best regards,

Kirill

 

www.jedox.com
Kirill Rajbhandary
Senior Software Engineer
<a href="tel:+49%20761%2015147%20237" target="_blank" id="LPlnk689713" style="text-decoration:none;color:#807F7F;">+49 761 15147 237
Jedox AG
Bismarckallee 7a
79098
Freiburg im Breisgau
Germany
Executive Board: Florian Winterstein (Chairman), Bernd Eisenblätter, Maximilian Prinz zu Hohenlohe‑Waldenburg
​Supervisory Board: Bernhard Wöbker (Chairman), Curt Gunsenheimer, Thilo Schmid
​Place of Business: Amtsgericht Freiburg HRB 702118

​​[TM.V06112018]
Reply | Threaded
Open this post in threaded view
|

RE: POI memory usage degradation

Kirill Rajbhandary
Dear POI Users Group,

Unfortunately I did not subscribe to POI Users List and did not get any information about the raised issue.
Now I have a subscription.
Could you please update me regarding the issue?

Best regards,
Kirill


From: Kirill Rajbhandary
Sent: Wednesday, 5 June, 2019 09:49
To: [hidden email]
Subject: POI memory usage degradation

Dear POI Users Group,

Unfortunately, I found out that the latest POI library poi-ooxml with versions 4.0.1, 4.1.0 consumes much more memory than the version 3.17 when it is required to write information to already existing large Excel file.
It requires approximately 6GB Java Heap Space to add these rows using poi-ooxml library with version 3.17 but it is not enough even twice more (12GB) with version 4.0.1 or 4.1.0. Besides, the performance is much worse with the latest versions of poi-ooxml. OpenJDK 11.0.02 was used to run this example.
In attachment the corresponding example with ExcelWriter class can be found that tries to add only 5 rows to already existing sheet "Sheet2" having only header row with 16 columns. The first sheet "Sheet1" of this file already contains 240K rows with 23 columns. Zipped ExcelWriter  with Test-Template.xslx can be downloaded from the next link:

https://drive.google.com/file/d/1vq6WvMv5mSFUzyJinN9aoYRBStxZuJCq/view

To run ExcelWriter main method, please specify the full path to Test-Template.xlsx and result file path. To change poi-ooxml version, it requires to change it in pom.xml, rebuild the project and run it again.

Could you please advise why it became worse comparing with the old library version poi-ooxml-3.17.jar and how it could be improved?

Best regards,
Kirill

www.jedox.com
Kirill Rajbhandary
Senior Software Engineer
<a href="tel:+49%20761%2015147%20237" target="_blank" id="LPlnk689713" style="text-decoration:none;color:#807F7F;">+49 761 15147 237
Jedox AG
Bismarckallee 7a
79098
Freiburg im Breisgau
Germany
Executive Board: Florian Winterstein (Chairman), Bernd Eisenblätter, Maximilian Prinz zu Hohenlohe‑Waldenburg
​Supervisory Board: Bernhard Wöbker (Chairman), Curt Gunsenheimer, Thilo Schmid
​Place of Business: Amtsgericht Freiburg HRB 702118

​​[TM.V06112018]
Reply | Threaded
Open this post in threaded view
|

Re: POI memory usage degradation

Mark Murphy
I see no issue in the Bugzilla. Without an issue there, the chances of it
being worked on or even considered are poor at best.

On Fri, Jul 12, 2019 at 9:29 AM Kirill Rajbhandary <
[hidden email]> wrote:

> Dear POI Users Group,
>
> Unfortunately I did not subscribe to POI Users List and did not get any
> information about the raised issue.
> Now I have a subscription.
> Could you please update me regarding the issue?
>
> Best regards,
> Kirill
>
>
> From: Kirill Rajbhandary
> Sent: Wednesday, 5 June, 2019 09:49
> To: [hidden email]
> Subject: POI memory usage degradation
>
> Dear POI Users Group,
>
> Unfortunately, I found out that the latest POI library poi-ooxml with
> versions 4.0.1, 4.1.0 consumes much more memory than the version 3.17 when
> it is required to write information to already existing large Excel file.
> It requires approximately 6GB Java Heap Space to add these rows using
> poi-ooxml library with version 3.17 but it is not enough even twice more
> (12GB) with version 4.0.1 or 4.1.0. Besides, the performance is much worse
> with the latest versions of poi-ooxml. OpenJDK 11.0.02 was used to run this
> example.
> In attachment the corresponding example with ExcelWriter class can be
> found that tries to add only 5 rows to already existing sheet "Sheet2"
> having only header row with 16 columns. The first sheet "Sheet1" of this
> file already contains 240K rows with 23 columns. Zipped ExcelWriter  with
> Test-Template.xslx can be downloaded from the next link:
>
> https://drive.google.com/file/d/1vq6WvMv5mSFUzyJinN9aoYRBStxZuJCq/view
>
> To run ExcelWriter main method, please specify the full path to
> Test-Template.xlsx and result file path. To change poi-ooxml version, it
> requires to change it in pom.xml, rebuild the project and run it again.
>
> Could you please advise why it became worse comparing with the old library
> version poi-ooxml-3.17.jar and how it could be improved?
>
> Best regards,
> Kirill
>
> <https://www.jedox.com/de/cpm-software/testversion/>
> *www.jedox.com* <https://www.jedox.com/de/>
> Kirill Rajbhandary​
> Senior Software Engineer
> *+49 761 15147 237* <+49%20761%2015147%20237>
> Jedox AG
> Bismarckallee 7a
> 79098
> Freiburg im Breisgau
> Germany
>
> Executive Board: Florian Winterstein (Chairman), Bernd Eisenblätter, Maximilian Prinz zu Hohenlohe‑Waldenburg
>
> ​Supervisory Board: Bernhard Wöbker (Chairman), Curt Gunsenheimer, Thilo Schmid
> ​Place of Business: Amtsgericht Freiburg HRB 702118
> ​
> ​​[TM.V06112018]
>
Reply | Threaded
Open this post in threaded view
|

Re: POI memory usage degradation

Mark Murphy
Maybe you could take a look and see if you can narrow down what the issues
are. Then maybe you could contribute a patch?

On Fri, Jul 12, 2019 at 11:10 AM Mark Murphy <[hidden email]> wrote:

> I see no issue in the Bugzilla. Without an issue there, the chances of it
> being worked on or even considered are poor at best.
>
> On Fri, Jul 12, 2019 at 9:29 AM Kirill Rajbhandary <
> [hidden email]> wrote:
>
>> Dear POI Users Group,
>>
>> Unfortunately I did not subscribe to POI Users List and did not get any
>> information about the raised issue.
>> Now I have a subscription.
>> Could you please update me regarding the issue?
>>
>> Best regards,
>> Kirill
>>
>>
>> From: Kirill Rajbhandary
>> Sent: Wednesday, 5 June, 2019 09:49
>> To: [hidden email]
>> Subject: POI memory usage degradation
>>
>> Dear POI Users Group,
>>
>> Unfortunately, I found out that the latest POI library poi-ooxml with
>> versions 4.0.1, 4.1.0 consumes much more memory than the version 3.17 when
>> it is required to write information to already existing large Excel file.
>> It requires approximately 6GB Java Heap Space to add these rows using
>> poi-ooxml library with version 3.17 but it is not enough even twice more
>> (12GB) with version 4.0.1 or 4.1.0. Besides, the performance is much worse
>> with the latest versions of poi-ooxml. OpenJDK 11.0.02 was used to run this
>> example.
>> In attachment the corresponding example with ExcelWriter class can be
>> found that tries to add only 5 rows to already existing sheet "Sheet2"
>> having only header row with 16 columns. The first sheet "Sheet1" of this
>> file already contains 240K rows with 23 columns. Zipped ExcelWriter  with
>> Test-Template.xslx can be downloaded from the next link:
>>
>> https://drive.google.com/file/d/1vq6WvMv5mSFUzyJinN9aoYRBStxZuJCq/view
>>
>> To run ExcelWriter main method, please specify the full path to
>> Test-Template.xlsx and result file path. To change poi-ooxml version, it
>> requires to change it in pom.xml, rebuild the project and run it again.
>>
>> Could you please advise why it became worse comparing with the old
>> library version poi-ooxml-3.17.jar and how it could be improved?
>>
>> Best regards,
>> Kirill
>>
>> <https://www.jedox.com/de/cpm-software/testversion/>
>> *www.jedox.com* <https://www.jedox.com/de/>
>> Kirill Rajbhandary​
>> Senior Software Engineer
>> *+49 761 15147 237* <+49%20761%2015147%20237>
>> Jedox AG
>> Bismarckallee 7a
>> 79098
>> Freiburg im Breisgau
>> Germany
>>
>> Executive Board: Florian Winterstein (Chairman), Bernd Eisenblätter, Maximilian Prinz zu Hohenlohe‑Waldenburg
>>
>> ​Supervisory Board: Bernhard Wöbker (Chairman), Curt Gunsenheimer, Thilo Schmid
>> ​Place of Business: Amtsgericht Freiburg HRB 702118
>> ​
>> ​​[TM.V06112018]
>>
>
Reply | Threaded
Open this post in threaded view
|

RE: POI memory usage degradation

Kirill R
In reply to this post by Kirill Rajbhandary

Dear POI Users Group,


I created a smaller example regarding this issue excel-writer.zip with readme.md inside how to run it. Additionally I made a heap memory dump after OutOfMemoryError has been thrown. I assume that the root cause of the issue is in the dependency xmlbeans-3.1.0.jar which was updated as well and according to the heap dump the most memory is used by org.apache.xmlbeans.impl.store.Xobj$AttrXobj. Here is the stack trace of the error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
at java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
at org.apache.poi.openxml4j.opc.internal.MemoryPackagePartOutputStream.write(MemoryPackagePartOutputStream.java:88)
at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:590)
at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2544)
at org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:223)
at org.apache.poi.xssf.model.SharedStringsTable.writeTo(SharedStringsTable.java:279)
at org.apache.poi.xssf.model.SharedStringsTable.commit(SharedStringsTable.java:286)
at org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:463)
at org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:468)
at org.apache.poi.ooxml.POIXMLDocument.write(POIXMLDocument.java:236)
at org.apache.poi.xssf.streaming.SXSSFWorkbook.write(SXSSFWorkbook.java:943)
at com.excel.writer.ExcelWriter.write(ExcelWriter.java:48)
at com.excel.writer.ExcelWriter.main(ExcelWriter.java:74)


For the poi-ooxml with version 3.17 and its dependency xmlbeans-2.6.0.jar, memory usage was twice better and it is enough to set 512MB to run this example, whereas using the latest 4.1.0 version, it requires more than twice more memory (1.1GB ) to be executed without 

OutOfMemoryError. It is not critical for small files like in this new example, but it becomes critical if the file is much bigger and required 10GB or even much more memory to be written.

Best regards,
Kirill
 

-----Original Message-----
From: Mark Murphy <[hidden email]>
Sent: Friday, 12 July, 2019 17:12
To: POI Users List <[hidden email]>
Subject: Re: POI memory usage degradation

 

Maybe you could take a look and see if you can narrow down what the issues are. Then maybe you could contribute a patch?

 

On Fri, Jul 12, 2019 at 11:10 AM Mark Murphy <[hidden email]> wrote:

 

> I see no issue in the Bugzilla. Without an issue there, the chances of

> it being worked on or even considered are poor at best.

> On Fri, Jul 12, 2019 at 9:29 AM Kirill wrote:

>> Dear POI Users Group,

>> 

>> Unfortunately I did not subscribe to POI Users List and did not get

>> any information about the raised issue.

>> Now I have a subscription.

>> Could you please update me regarding the issue?

>> 

>> Best regards,

>> Kirill



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

excel-writer.zip (508K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: POI memory usage degradation

Dominik Stadler
Hi,

after some "git bisecting" this problem, it turns out that likely change
https://svn.apache.org/viewvc?view=revision&revision=1825277 is somehow
increasing memory usage when writing the document. It is not caused by
XMLBeans per se, but how it is used at this point and how XMLBeans handles
arrays.

We need to look if we can do this differently, naturally this change tried
to fix another bug, which we don't want to re-introduce again here as well.

Regards... Dominik.

On Tue, Jul 16, 2019 at 1:36 PM Kirill R <[hidden email]> wrote:

> Dear POI Users Group,
>
>
> I created a smaller example regarding this issue excel-writer.zip with
> readme.md inside how to run it. Additionally I made a heap memory dump
> after OutOfMemoryError has been thrown. I assume that the root cause of the
> issue is in the dependency xmlbeans-3.1.0.jar which was updated as well and
> according to the heap dump the most memory is used by
> org.apache.xmlbeans.impl.store.Xobj$AttrXobj. Here is the stack trace of
> the error:
>
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
> at
> java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
> at
> java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
> at
> java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
> at
> org.apache.poi.openxml4j.opc.internal.MemoryPackagePartOutputStream.write(MemoryPackagePartOutputStream.java:88)
> at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:590)
> at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2544)
> at
> org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:223)
> at
> org.apache.poi.xssf.model.SharedStringsTable.writeTo(SharedStringsTable.java:279)
> at
> org.apache.poi.xssf.model.SharedStringsTable.commit(SharedStringsTable.java:286)
> at
> org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:463)
> at
> org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:468)
> at org.apache.poi.ooxml.POIXMLDocument.write(POIXMLDocument.java:236)
> at
> org.apache.poi.xssf.streaming.SXSSFWorkbook.write(SXSSFWorkbook.java:943)
> at com.excel.writer.ExcelWriter.write(ExcelWriter.java:48)
> at com.excel.writer.ExcelWriter.main(ExcelWriter.java:74)
>
>
> For the poi-ooxml with version 3.17 and its dependency xmlbeans-2.6.0.jar,
> memory usage was twice better and it is enough to set 512MB to run this
> example, whereas using the latest 4.1.0 version, it requires more than
> twice more memory (1.1GB ) to be executed without
>
> OutOfMemoryError. It is not critical for small files like in this new
> example, but it becomes critical if the file is much bigger and required
> 10GB or even much more memory to be written.
>
> Best regards,
> Kirill
>
>
> -----Original Message-----
> From: Mark Murphy <[hidden email]>
> Sent: Friday, 12 July, 2019 17:12
> To: POI Users List <[hidden email]>
> Subject: Re: POI memory usage degradation
>
>
>
> Maybe you could take a look and see if you can narrow down what the issues
> are. Then maybe you could contribute a patch?
>
>
>
> On Fri, Jul 12, 2019 at 11:10 AM Mark Murphy <[hidden email]>
> wrote:
>
>
>
> > I see no issue in the Bugzilla. Without an issue there, the chances of
>
> > it being worked on or even considered are poor at best.
>
> >
>
> > On Fri, Jul 12, 2019 at 9:29 AM Kirill wrote:
>
> >
>
> >> Dear POI Users Group,
>
> >>
>
> >> Unfortunately I did not subscribe to POI Users List and did not get
>
> >> any information about the raised issue.
>
> >> Now I have a subscription.
>
> >> Could you please update me regarding the issue?
>
> >>
>
> >> Best regards,
>
> >> Kirill
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: POI memory usage degradation

Kirill R
Dear Domenik,

Could you please create the corresponding ticket for this?

Best regards,
Kirill


On Sun, Aug 4, 2019 at 8:07 PM Dominik Stadler <[hidden email]>
wrote:

> Hi,
>
> after some "git bisecting" this problem, it turns out that likely change
> https://svn.apache.org/viewvc?view=revision&revision=1825277 is somehow
> increasing memory usage when writing the document. It is not caused by
> XMLBeans per se, but how it is used at this point and how XMLBeans handles
> arrays.
>
> We need to look if we can do this differently, naturally this change tried
> to fix another bug, which we don't want to re-introduce again here as well.
>
> Regards... Dominik.
>
> On Tue, Jul 16, 2019 at 1:36 PM Kirill R <[hidden email]> wrote:
>
> > Dear POI Users Group,
> >
> >
> > I created a smaller example regarding this issue excel-writer.zip with
> > readme.md inside how to run it. Additionally I made a heap memory dump
> > after OutOfMemoryError has been thrown. I assume that the root cause of
> the
> > issue is in the dependency xmlbeans-3.1.0.jar which was updated as well
> and
> > according to the heap dump the most memory is used by
> > org.apache.xmlbeans.impl.store.Xobj$AttrXobj. Here is the stack trace of
> > the error:
> >
> > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> > at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
> > at
> > java.base/java.io
> .ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
> > at
> > java.base/java.io
> .ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
> > at
> > java.base/java.io
> .ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
> > at
> >
> org.apache.poi.openxml4j.opc.internal.MemoryPackagePartOutputStream.write(MemoryPackagePartOutputStream.java:88)
> > at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:590)
> > at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2544)
> > at
> >
> org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:223)
> > at
> >
> org.apache.poi.xssf.model.SharedStringsTable.writeTo(SharedStringsTable.java:279)
> > at
> >
> org.apache.poi.xssf.model.SharedStringsTable.commit(SharedStringsTable.java:286)
> > at
> >
> org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:463)
> > at
> >
> org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:468)
> > at org.apache.poi.ooxml.POIXMLDocument.write(POIXMLDocument.java:236)
> > at
> > org.apache.poi.xssf.streaming.SXSSFWorkbook.write(SXSSFWorkbook.java:943)
> > at com.excel.writer.ExcelWriter.write(ExcelWriter.java:48)
> > at com.excel.writer.ExcelWriter.main(ExcelWriter.java:74)
> >
> >
> > For the poi-ooxml with version 3.17 and its dependency
> xmlbeans-2.6.0.jar,
> > memory usage was twice better and it is enough to set 512MB to run this
> > example, whereas using the latest 4.1.0 version, it requires more than
> > twice more memory (1.1GB ) to be executed without
> >
> > OutOfMemoryError. It is not critical for small files like in this new
> > example, but it becomes critical if the file is much bigger and required
> > 10GB or even much more memory to be written.
> >
> > Best regards,
> > Kirill
> >
> >
> > -----Original Message-----
> > From: Mark Murphy <[hidden email]>
> > Sent: Friday, 12 July, 2019 17:12
> > To: POI Users List <[hidden email]>
> > Subject: Re: POI memory usage degradation
> >
> >
> >
> > Maybe you could take a look and see if you can narrow down what the
> issues
> > are. Then maybe you could contribute a patch?
> >
> >
> >
> > On Fri, Jul 12, 2019 at 11:10 AM Mark Murphy <[hidden email]>
> > wrote:
> >
> >
> >
> > > I see no issue in the Bugzilla. Without an issue there, the chances of
> >
> > > it being worked on or even considered are poor at best.
> >
> > >
> >
> > > On Fri, Jul 12, 2019 at 9:29 AM Kirill wrote:
> >
> > >
> >
> > >> Dear POI Users Group,
> >
> > >>
> >
> > >> Unfortunately I did not subscribe to POI Users List and did not get
> >
> > >> any information about the raised issue.
> >
> > >> Now I have a subscription.
> >
> > >> Could you please update me regarding the issue?
> >
> > >>
> >
> > >> Best regards,
> >
> > >> Kirill
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
>
Reply | Threaded
Open this post in threaded view
|

Re: POI memory usage degradation

Dominik Stadler
Hi,

this should be fixed now again, see
https://bz.apache.org/bugzilla/show_bug.cgi?id=63657, please test it with
pre-builds from
https://builds.apache.org/view/P/view/POI/job/POI-DSL-1.8/lastSuccessfulBuild/artifact/build/dist/
so we are sure it fixes your use-case properly.

Dominik.

On Mon, Aug 5, 2019 at 10:13 AM Kirill <[hidden email]> wrote:

> Dear Domenik,
>
> Could you please create the corresponding ticket for this?
>
> Best regards,
> Kirill
>
>
> On Sun, Aug 4, 2019 at 8:07 PM Dominik Stadler <[hidden email]>
> wrote:
>
>> Hi,
>>
>> after some "git bisecting" this problem, it turns out that likely change
>> https://svn.apache.org/viewvc?view=revision&revision=1825277 is somehow
>> increasing memory usage when writing the document. It is not caused by
>> XMLBeans per se, but how it is used at this point and how XMLBeans handles
>> arrays.
>>
>> We need to look if we can do this differently, naturally this change tried
>> to fix another bug, which we don't want to re-introduce again here as
>> well.
>>
>> Regards... Dominik.
>>
>> On Tue, Jul 16, 2019 at 1:36 PM Kirill R <[hidden email]> wrote:
>>
>> > Dear POI Users Group,
>> >
>> >
>> > I created a smaller example regarding this issue excel-writer.zip with
>> > readme.md inside how to run it. Additionally I made a heap memory dump
>> > after OutOfMemoryError has been thrown. I assume that the root cause of
>> the
>> > issue is in the dependency xmlbeans-3.1.0.jar which was updated as well
>> and
>> > according to the heap dump the most memory is used by
>> > org.apache.xmlbeans.impl.store.Xobj$AttrXobj. Here is the stack trace of
>> > the error:
>> >
>> > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>> > at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
>> > at
>> > java.base/java.io
>> .ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
>> > at
>> > java.base/java.io
>> .ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
>> > at
>> > java.base/java.io
>> .ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
>> > at
>> >
>> org.apache.poi.openxml4j.opc.internal.MemoryPackagePartOutputStream.write(MemoryPackagePartOutputStream.java:88)
>> > at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:590)
>> > at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2544)
>> > at
>> >
>> org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:223)
>> > at
>> >
>> org.apache.poi.xssf.model.SharedStringsTable.writeTo(SharedStringsTable.java:279)
>> > at
>> >
>> org.apache.poi.xssf.model.SharedStringsTable.commit(SharedStringsTable.java:286)
>> > at
>> >
>> org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:463)
>> > at
>> >
>> org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:468)
>> > at org.apache.poi.ooxml.POIXMLDocument.write(POIXMLDocument.java:236)
>> > at
>> >
>> org.apache.poi.xssf.streaming.SXSSFWorkbook.write(SXSSFWorkbook.java:943)
>> > at com.excel.writer.ExcelWriter.write(ExcelWriter.java:48)
>> > at com.excel.writer.ExcelWriter.main(ExcelWriter.java:74)
>> >
>> >
>> > For the poi-ooxml with version 3.17 and its dependency
>> xmlbeans-2.6.0.jar,
>> > memory usage was twice better and it is enough to set 512MB to run this
>> > example, whereas using the latest 4.1.0 version, it requires more than
>> > twice more memory (1.1GB ) to be executed without
>> >
>> > OutOfMemoryError. It is not critical for small files like in this new
>> > example, but it becomes critical if the file is much bigger and required
>> > 10GB or even much more memory to be written.
>> >
>> > Best regards,
>> > Kirill
>> >
>> >
>> > -----Original Message-----
>> > From: Mark Murphy <[hidden email]>
>> > Sent: Friday, 12 July, 2019 17:12
>> > To: POI Users List <[hidden email]>
>> > Subject: Re: POI memory usage degradation
>> >
>> >
>> >
>> > Maybe you could take a look and see if you can narrow down what the
>> issues
>> > are. Then maybe you could contribute a patch?
>> >
>> >
>> >
>> > On Fri, Jul 12, 2019 at 11:10 AM Mark Murphy <[hidden email]>
>> > wrote:
>> >
>> >
>> >
>> > > I see no issue in the Bugzilla. Without an issue there, the chances of
>> >
>> > > it being worked on or even considered are poor at best.
>> >
>> > >
>> >
>> > > On Fri, Jul 12, 2019 at 9:29 AM Kirill wrote:
>> >
>> > >
>> >
>> > >> Dear POI Users Group,
>> >
>> > >>
>> >
>> > >> Unfortunately I did not subscribe to POI Users List and did not get
>> >
>> > >> any information about the raised issue.
>> >
>> > >> Now I have a subscription.
>> >
>> > >> Could you please update me regarding the issue?
>> >
>> > >>
>> >
>> > >> Best regards,
>> >
>> > >> Kirill
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: [hidden email]
>> > For additional commands, e-mail: [hidden email]
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: POI memory usage degradation

Kirill R
Hi,

Unfortunately the same OutOfMemoryError is thrown with the same stacktrace
if I try to decrease memory allocation.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
at
java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
at
java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
at
java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
at
org.apache.poi.openxml4j.opc.internal.MemoryPackagePartOutputStream.write(MemoryPackagePartOutputStream.java:88)
at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:590)
at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2544)
at
org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:223)
at org.apache.poi.xssf.usermodel.XSSFSheet.write(XSSFSheet.java:3651)
at org.apache.poi.xssf.usermodel.XSSFSheet.commit(XSSFSheet.java:3572)
at
org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:463)
at
org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:468)
at org.apache.poi.ooxml.POIXMLDocument.write(POIXMLDocument.java:236)
at org.apache.poi.xssf.streaming.SXSSFWorkbook.write(SXSSFWorkbook.java:943)

Best regards,
Kirill


On Tue, Aug 13, 2019 at 4:17 PM Dominik Stadler <[hidden email]>
wrote:

> Hi,
>
> this should be fixed now again, see
> https://bz.apache.org/bugzilla/show_bug.cgi?id=63657, please test it with
> pre-builds from
> https://builds.apache.org/view/P/view/POI/job/POI-DSL-1.8/lastSuccessfulBuild/artifact/build/dist/
> so we are sure it fixes your use-case properly.
>
> Dominik.
>
> On Mon, Aug 5, 2019 at 10:13 AM Kirill <[hidden email]> wrote:
>
>> Dear Domenik,
>>
>> Could you please create the corresponding ticket for this?
>>
>> Best regards,
>> Kirill
>>
>>
>> On Sun, Aug 4, 2019 at 8:07 PM Dominik Stadler <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> after some "git bisecting" this problem, it turns out that likely change
>>> https://svn.apache.org/viewvc?view=revision&revision=1825277 is somehow
>>> increasing memory usage when writing the document. It is not caused by
>>> XMLBeans per se, but how it is used at this point and how XMLBeans
>>> handles
>>> arrays.
>>>
>>> We need to look if we can do this differently, naturally this change
>>> tried
>>> to fix another bug, which we don't want to re-introduce again here as
>>> well.
>>>
>>> Regards... Dominik.
>>>
>>> On Tue, Jul 16, 2019 at 1:36 PM Kirill R <[hidden email]> wrote:
>>>
>>> > Dear POI Users Group,
>>> >
>>> >
>>> > I created a smaller example regarding this issue excel-writer.zip with
>>> > readme.md inside how to run it. Additionally I made a heap memory dump
>>> > after OutOfMemoryError has been thrown. I assume that the root cause
>>> of the
>>> > issue is in the dependency xmlbeans-3.1.0.jar which was updated as
>>> well and
>>> > according to the heap dump the most memory is used by
>>> > org.apache.xmlbeans.impl.store.Xobj$AttrXobj. Here is the stack trace
>>> of
>>> > the error:
>>> >
>>> > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>>> > at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
>>> > at
>>> > java.base/java.io
>>> .ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
>>> > at
>>> > java.base/java.io
>>> .ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
>>> > at
>>> > java.base/java.io
>>> .ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
>>> > at
>>> >
>>> org.apache.poi.openxml4j.opc.internal.MemoryPackagePartOutputStream.write(MemoryPackagePartOutputStream.java:88)
>>> > at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:590)
>>> > at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2544)
>>> > at
>>> >
>>> org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:223)
>>> > at
>>> >
>>> org.apache.poi.xssf.model.SharedStringsTable.writeTo(SharedStringsTable.java:279)
>>> > at
>>> >
>>> org.apache.poi.xssf.model.SharedStringsTable.commit(SharedStringsTable.java:286)
>>> > at
>>> >
>>> org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:463)
>>> > at
>>> >
>>> org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:468)
>>> > at org.apache.poi.ooxml.POIXMLDocument.write(POIXMLDocument.java:236)
>>> > at
>>> >
>>> org.apache.poi.xssf.streaming.SXSSFWorkbook.write(SXSSFWorkbook.java:943)
>>> > at com.excel.writer.ExcelWriter.write(ExcelWriter.java:48)
>>> > at com.excel.writer.ExcelWriter.main(ExcelWriter.java:74)
>>> >
>>> >
>>> > For the poi-ooxml with version 3.17 and its dependency
>>> xmlbeans-2.6.0.jar,
>>> > memory usage was twice better and it is enough to set 512MB to run this
>>> > example, whereas using the latest 4.1.0 version, it requires more than
>>> > twice more memory (1.1GB ) to be executed without
>>> >
>>> > OutOfMemoryError. It is not critical for small files like in this new
>>> > example, but it becomes critical if the file is much bigger and
>>> required
>>> > 10GB or even much more memory to be written.
>>> >
>>> > Best regards,
>>> > Kirill
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: Mark Murphy <[hidden email]>
>>> > Sent: Friday, 12 July, 2019 17:12
>>> > To: POI Users List <[hidden email]>
>>> > Subject: Re: POI memory usage degradation
>>> >
>>> >
>>> >
>>> > Maybe you could take a look and see if you can narrow down what the
>>> issues
>>> > are. Then maybe you could contribute a patch?
>>> >
>>> >
>>> >
>>> > On Fri, Jul 12, 2019 at 11:10 AM Mark Murphy <[hidden email]>
>>> > wrote:
>>> >
>>> >
>>> >
>>> > > I see no issue in the Bugzilla. Without an issue there, the chances
>>> of
>>> >
>>> > > it being worked on or even considered are poor at best.
>>> >
>>> > >
>>> >
>>> > > On Fri, Jul 12, 2019 at 9:29 AM Kirill wrote:
>>> >
>>> > >
>>> >
>>> > >> Dear POI Users Group,
>>> >
>>> > >>
>>> >
>>> > >> Unfortunately I did not subscribe to POI Users List and did not get
>>> >
>>> > >> any information about the raised issue.
>>> >
>>> > >> Now I have a subscription.
>>> >
>>> > >> Could you please update me regarding the issue?
>>> >
>>> > >>
>>> >
>>> > >> Best regards,
>>> >
>>> > >> Kirill
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: [hidden email]
>>> > For additional commands, e-mail: [hidden email]
>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: POI memory usage degradation

Dominik Stadler
Stacktrace indicates that this is not the latest version from trunk, are
you sure you updated?

D.

On Thu, Aug 22, 2019 at 1:16 PM Kirill <[hidden email]> wrote:

> Hi,
>
> Unfortunately the same OutOfMemoryError is thrown with the same stacktrace
> if I try to decrease memory allocation.
>
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
> at
> java.base/java.io
> .ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
> at
> java.base/java.io
> .ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
> at
> java.base/java.io
> .ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
> at
>
> org.apache.poi.openxml4j.opc.internal.MemoryPackagePartOutputStream.write(MemoryPackagePartOutputStream.java:88)
> at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:590)
> at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2544)
> at
> org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:223)
> at org.apache.poi.xssf.usermodel.XSSFSheet.write(XSSFSheet.java:3651)
> at org.apache.poi.xssf.usermodel.XSSFSheet.commit(XSSFSheet.java:3572)
> at
> org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:463)
> at
> org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:468)
> at org.apache.poi.ooxml.POIXMLDocument.write(POIXMLDocument.java:236)
> at
> org.apache.poi.xssf.streaming.SXSSFWorkbook.write(SXSSFWorkbook.java:943)
>
> Best regards,
> Kirill
>
>
> On Tue, Aug 13, 2019 at 4:17 PM Dominik Stadler <[hidden email]>
> wrote:
>
> > Hi,
> >
> > this should be fixed now again, see
> > https://bz.apache.org/bugzilla/show_bug.cgi?id=63657, please test it
> with
> > pre-builds from
> >
> https://builds.apache.org/view/P/view/POI/job/POI-DSL-1.8/lastSuccessfulBuild/artifact/build/dist/
> > so we are sure it fixes your use-case properly.
> >
> > Dominik.
> >
> > On Mon, Aug 5, 2019 at 10:13 AM Kirill <[hidden email]> wrote:
> >
> >> Dear Domenik,
> >>
> >> Could you please create the corresponding ticket for this?
> >>
> >> Best regards,
> >> Kirill
> >>
> >>
> >> On Sun, Aug 4, 2019 at 8:07 PM Dominik Stadler <[hidden email]>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> after some "git bisecting" this problem, it turns out that likely
> change
> >>> https://svn.apache.org/viewvc?view=revision&revision=1825277 is
> somehow
> >>> increasing memory usage when writing the document. It is not caused by
> >>> XMLBeans per se, but how it is used at this point and how XMLBeans
> >>> handles
> >>> arrays.
> >>>
> >>> We need to look if we can do this differently, naturally this change
> >>> tried
> >>> to fix another bug, which we don't want to re-introduce again here as
> >>> well.
> >>>
> >>> Regards... Dominik.
> >>>
> >>> On Tue, Jul 16, 2019 at 1:36 PM Kirill R <[hidden email]> wrote:
> >>>
> >>> > Dear POI Users Group,
> >>> >
> >>> >
> >>> > I created a smaller example regarding this issue excel-writer.zip
> with
> >>> > readme.md inside how to run it. Additionally I made a heap memory
> dump
> >>> > after OutOfMemoryError has been thrown. I assume that the root cause
> >>> of the
> >>> > issue is in the dependency xmlbeans-3.1.0.jar which was updated as
> >>> well and
> >>> > according to the heap dump the most memory is used by
> >>> > org.apache.xmlbeans.impl.store.Xobj$AttrXobj. Here is the stack trace
> >>> of
> >>> > the error:
> >>> >
> >>> > Exception in thread "main" java.lang.OutOfMemoryError: Java heap
> space
> >>> > at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
> >>> > at
> >>> > java.base/java.io
> >>> .ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
> >>> > at
> >>> > java.base/java.io
> >>> .ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
> >>> > at
> >>> > java.base/java.io
> >>> .ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
> >>> > at
> >>> >
> >>>
> org.apache.poi.openxml4j.opc.internal.MemoryPackagePartOutputStream.write(MemoryPackagePartOutputStream.java:88)
> >>> > at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:590)
> >>> > at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2544)
> >>> > at
> >>> >
> >>>
> org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:223)
> >>> > at
> >>> >
> >>>
> org.apache.poi.xssf.model.SharedStringsTable.writeTo(SharedStringsTable.java:279)
> >>> > at
> >>> >
> >>>
> org.apache.poi.xssf.model.SharedStringsTable.commit(SharedStringsTable.java:286)
> >>> > at
> >>> >
> >>>
> org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:463)
> >>> > at
> >>> >
> >>>
> org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:468)
> >>> > at org.apache.poi.ooxml.POIXMLDocument.write(POIXMLDocument.java:236)
> >>> > at
> >>> >
> >>>
> org.apache.poi.xssf.streaming.SXSSFWorkbook.write(SXSSFWorkbook.java:943)
> >>> > at com.excel.writer.ExcelWriter.write(ExcelWriter.java:48)
> >>> > at com.excel.writer.ExcelWriter.main(ExcelWriter.java:74)
> >>> >
> >>> >
> >>> > For the poi-ooxml with version 3.17 and its dependency
> >>> xmlbeans-2.6.0.jar,
> >>> > memory usage was twice better and it is enough to set 512MB to run
> this
> >>> > example, whereas using the latest 4.1.0 version, it requires more
> than
> >>> > twice more memory (1.1GB ) to be executed without
> >>> >
> >>> > OutOfMemoryError. It is not critical for small files like in this new
> >>> > example, but it becomes critical if the file is much bigger and
> >>> required
> >>> > 10GB or even much more memory to be written.
> >>> >
> >>> > Best regards,
> >>> > Kirill
> >>> >
> >>> >
> >>> > -----Original Message-----
> >>> > From: Mark Murphy <[hidden email]>
> >>> > Sent: Friday, 12 July, 2019 17:12
> >>> > To: POI Users List <[hidden email]>
> >>> > Subject: Re: POI memory usage degradation
> >>> >
> >>> >
> >>> >
> >>> > Maybe you could take a look and see if you can narrow down what the
> >>> issues
> >>> > are. Then maybe you could contribute a patch?
> >>> >
> >>> >
> >>> >
> >>> > On Fri, Jul 12, 2019 at 11:10 AM Mark Murphy <[hidden email]>
> >>> > wrote:
> >>> >
> >>> >
> >>> >
> >>> > > I see no issue in the Bugzilla. Without an issue there, the chances
> >>> of
> >>> >
> >>> > > it being worked on or even considered are poor at best.
> >>> >
> >>> > >
> >>> >
> >>> > > On Fri, Jul 12, 2019 at 9:29 AM Kirill wrote:
> >>> >
> >>> > >
> >>> >
> >>> > >> Dear POI Users Group,
> >>> >
> >>> > >>
> >>> >
> >>> > >> Unfortunately I did not subscribe to POI Users List and did not
> get
> >>> >
> >>> > >> any information about the raised issue.
> >>> >
> >>> > >> Now I have a subscription.
> >>> >
> >>> > >> Could you please update me regarding the issue?
> >>> >
> >>> > >>
> >>> >
> >>> > >> Best regards,
> >>> >
> >>> > >> Kirill
> >>> >
> >>> > ---------------------------------------------------------------------
> >>> > To unsubscribe, e-mail: [hidden email]
> >>> > For additional commands, e-mail: [hidden email]
> >>>
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: POI memory usage degradation

Kirill R
Hi,

I am sorry, I indeed missed to update some other libraries. After updating
all libraries, the issue is no longer reproduced.
Thank you for the fix. Will it be included in the next 4.1.1 version? When
is it approximately planned to release this new version?

Best regards,
Kirill


On Thu, Aug 22, 2019 at 6:00 PM Dominik Stadler <[hidden email]>
wrote:

> Stacktrace indicates that this is not the latest version from trunk, are
> you sure you updated?
>
> D.
>
> On Thu, Aug 22, 2019 at 1:16 PM Kirill <[hidden email]> wrote:
>
> > Hi,
> >
> > Unfortunately the same OutOfMemoryError is thrown with the same
> stacktrace
> > if I try to decrease memory allocation.
> >
> > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> > at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
> > at
> > java.base/java.io
> > .ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
> > at
> > java.base/java.io
> > .ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
> > at
> > java.base/java.io
> > .ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
> > at
> >
> >
> org.apache.poi.openxml4j.opc.internal.MemoryPackagePartOutputStream.write(MemoryPackagePartOutputStream.java:88)
> > at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:590)
> > at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2544)
> > at
> >
> org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:223)
> > at org.apache.poi.xssf.usermodel.XSSFSheet.write(XSSFSheet.java:3651)
> > at org.apache.poi.xssf.usermodel.XSSFSheet.commit(XSSFSheet.java:3572)
> > at
> >
> org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:463)
> > at
> >
> org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:468)
> > at org.apache.poi.ooxml.POIXMLDocument.write(POIXMLDocument.java:236)
> > at
> > org.apache.poi.xssf.streaming.SXSSFWorkbook.write(SXSSFWorkbook.java:943)
> >
> > Best regards,
> > Kirill
> >
> >
> > On Tue, Aug 13, 2019 at 4:17 PM Dominik Stadler <[hidden email]>
> > wrote:
> >
> > > Hi,
> > >
> > > this should be fixed now again, see
> > > https://bz.apache.org/bugzilla/show_bug.cgi?id=63657, please test it
> > with
> > > pre-builds from
> > >
> >
> https://builds.apache.org/view/P/view/POI/job/POI-DSL-1.8/lastSuccessfulBuild/artifact/build/dist/
> > > so we are sure it fixes your use-case properly.
> > >
> > > Dominik.
> > >
> > > On Mon, Aug 5, 2019 at 10:13 AM Kirill <[hidden email]> wrote:
> > >
> > >> Dear Domenik,
> > >>
> > >> Could you please create the corresponding ticket for this?
> > >>
> > >> Best regards,
> > >> Kirill
> > >>
> > >>
> > >> On Sun, Aug 4, 2019 at 8:07 PM Dominik Stadler <
> [hidden email]>
> > >> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> after some "git bisecting" this problem, it turns out that likely
> > change
> > >>> https://svn.apache.org/viewvc?view=revision&revision=1825277 is
> > somehow
> > >>> increasing memory usage when writing the document. It is not caused
> by
> > >>> XMLBeans per se, but how it is used at this point and how XMLBeans
> > >>> handles
> > >>> arrays.
> > >>>
> > >>> We need to look if we can do this differently, naturally this change
> > >>> tried
> > >>> to fix another bug, which we don't want to re-introduce again here as
> > >>> well.
> > >>>
> > >>> Regards... Dominik.
> > >>>
> > >>> On Tue, Jul 16, 2019 at 1:36 PM Kirill R <[hidden email]>
> wrote:
> > >>>
> > >>> > Dear POI Users Group,
> > >>> >
> > >>> >
> > >>> > I created a smaller example regarding this issue excel-writer.zip
> > with
> > >>> > readme.md inside how to run it. Additionally I made a heap memory
> > dump
> > >>> > after OutOfMemoryError has been thrown. I assume that the root
> cause
> > >>> of the
> > >>> > issue is in the dependency xmlbeans-3.1.0.jar which was updated as
> > >>> well and
> > >>> > according to the heap dump the most memory is used by
> > >>> > org.apache.xmlbeans.impl.store.Xobj$AttrXobj. Here is the stack
> trace
> > >>> of
> > >>> > the error:
> > >>> >
> > >>> > Exception in thread "main" java.lang.OutOfMemoryError: Java heap
> > space
> > >>> > at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
> > >>> > at
> > >>> > java.base/java.io
> > >>> .ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
> > >>> > at
> > >>> > java.base/java.io
> > >>> .ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
> > >>> > at
> > >>> > java.base/java.io
> > >>> .ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
> > >>> > at
> > >>> >
> > >>>
> >
> org.apache.poi.openxml4j.opc.internal.MemoryPackagePartOutputStream.write(MemoryPackagePartOutputStream.java:88)
> > >>> > at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:590)
> > >>> > at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2544)
> > >>> > at
> > >>> >
> > >>>
> >
> org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:223)
> > >>> > at
> > >>> >
> > >>>
> >
> org.apache.poi.xssf.model.SharedStringsTable.writeTo(SharedStringsTable.java:279)
> > >>> > at
> > >>> >
> > >>>
> >
> org.apache.poi.xssf.model.SharedStringsTable.commit(SharedStringsTable.java:286)
> > >>> > at
> > >>> >
> > >>>
> >
> org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:463)
> > >>> > at
> > >>> >
> > >>>
> >
> org.apache.poi.ooxml.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:468)
> > >>> > at
> org.apache.poi.ooxml.POIXMLDocument.write(POIXMLDocument.java:236)
> > >>> > at
> > >>> >
> > >>>
> > org.apache.poi.xssf.streaming.SXSSFWorkbook.write(SXSSFWorkbook.java:943)
> > >>> > at com.excel.writer.ExcelWriter.write(ExcelWriter.java:48)
> > >>> > at com.excel.writer.ExcelWriter.main(ExcelWriter.java:74)
> > >>> >
> > >>> >
> > >>> > For the poi-ooxml with version 3.17 and its dependency
> > >>> xmlbeans-2.6.0.jar,
> > >>> > memory usage was twice better and it is enough to set 512MB to run
> > this
> > >>> > example, whereas using the latest 4.1.0 version, it requires more
> > than
> > >>> > twice more memory (1.1GB ) to be executed without
> > >>> >
> > >>> > OutOfMemoryError. It is not critical for small files like in this
> new
> > >>> > example, but it becomes critical if the file is much bigger and
> > >>> required
> > >>> > 10GB or even much more memory to be written.
> > >>> >
> > >>> > Best regards,
> > >>> > Kirill
> > >>> >
> > >>> >
> > >>> > -----Original Message-----
> > >>> > From: Mark Murphy <[hidden email]>
> > >>> > Sent: Friday, 12 July, 2019 17:12
> > >>> > To: POI Users List <[hidden email]>
> > >>> > Subject: Re: POI memory usage degradation
> > >>> >
> > >>> >
> > >>> >
> > >>> > Maybe you could take a look and see if you can narrow down what the
> > >>> issues
> > >>> > are. Then maybe you could contribute a patch?
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Fri, Jul 12, 2019 at 11:10 AM Mark Murphy <
> [hidden email]>
> > >>> > wrote:
> > >>> >
> > >>> >
> > >>> >
> > >>> > > I see no issue in the Bugzilla. Without an issue there, the
> chances
> > >>> of
> > >>> >
> > >>> > > it being worked on or even considered are poor at best.
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > On Fri, Jul 12, 2019 at 9:29 AM Kirill wrote:
> > >>> >
> > >>> > >
> > >>> >
> > >>> > >> Dear POI Users Group,
> > >>> >
> > >>> > >>
> > >>> >
> > >>> > >> Unfortunately I did not subscribe to POI Users List and did not
> > get
> > >>> >
> > >>> > >> any information about the raised issue.
> > >>> >
> > >>> > >> Now I have a subscription.
> > >>> >
> > >>> > >> Could you please update me regarding the issue?
> > >>> >
> > >>> > >>
> > >>> >
> > >>> > >> Best regards,
> > >>> >
> > >>> > >> Kirill
> > >>> >
> > >>> >
> ---------------------------------------------------------------------
> > >>> > To unsubscribe, e-mail: [hidden email]
> > >>> > For additional commands, e-mail: [hidden email]
> > >>>
> > >>
> >
>